rarelens/docs/data.md

Data: what rarelens actually runs on

Everything below is public, peer-reviewed and consented for open redistribution. No patient data, no data access agreement, nothing that needs an application. These are the references to quote when showing the platform to someone.

Citations were verified against PubMed; each row links its DOI.

The demo slice

make data fetches two real files, chromosome 22 only (roughly 100 MB, minutes rather than hours):

File	What it is	Role
data/example.vcf.gz	GIAB HG002 (NA24385) v4.2.1 benchmark calls, GRCh38, chr22	the sample a scientist annotates
data/clinvar.chr22.vcf.gz	ClinVar, GRCh38, chr22	training labels, and the ClinVar column in the UI

HG002 is the NIST Genome in a Bottle Ashkenazi son, recruited through the Personal Genome Project, which consents participants to unrestricted public release. It is the reference genome the field benchmarks variant callers against, so it is both realistic and unambiguously shareable.

Datasets

Dataset	Used for	Access	Terms	Citation
ClinVar (GRCh38)	pathogenic/benign labels, ClinVar column	ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/	NCBI public domain	Landrum et al., Nucleic Acids Res 48(D1):D835–D844, 2020. 10.1093/nar/gkz972
Genome in a Bottle HG002 v4.2.1	the demo sample	ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/, s3://giab	open, no use restriction	Zook et al., Nat Biotechnol 37:561–566, 2019. 10.1038/s41587-019-0074-6
gnomAD v4	allele frequency feature and filter	gs://gcp-public-data--gnomad, s3://gnomad-public-us-east-1	free use, no restriction	Chen et al., Nature 625:92–100, 2024. 10.1038/s41586-023-06045-0; Karczewski et al., Nature 581:434–443, 2020. 10.1038/s41586-020-2308-7
1000 Genomes 30x	optional cohort/trio data	EBI FTP, s3://1000genomes	fully open, no access restriction	Byrska-Bishop et al., Cell 185(18):3426–3440.e19, 2022. 10.1016/j.cell.2022.08.004
MANE Select	one transcript per gene, if transcript choice ever matters	Ensembl/RefSeq	open	Morales et al., Nature 604:310–315, 2022. 10.1038/s41586-022-04558-8
Human Phenotype Ontology gene-to-phenotype	what the phenotype half of the ranking matches against (make hpo)	purl.obolibrary.org/obo/hp/hpoa/genes_to_phenotype.txt	free to use with attribution	Gargano et al., Nucleic Acids Res 52(D1):D1333–D1346, 2024. 10.1093/nar/gkad1005

Tools and scores

Tool	Role	Terms	Citation
Ensembl VEP 113	annotation (pipeline/modules/vep.nf)	Apache 2.0	McLaren et al., Genome Biol 17:122, 2016. 10.1186/s13059-016-0974-4
CADD	cadd_phred feature	free for non-commercial use; commercial licence required	Rentzsch et al., Nucleic Acids Res 47(D1):D886–D894, 2019. 10.1093/nar/gky1016; Schubach et al., Nucleic Acids Res 52(D1), 2024. 10.1093/nar/gkad989
AlphaMissense	am_pathogenicity feature	predictions moved to CC BY 4.0 in March 2024 (originally CC BY-NC-SA)	Cheng et al., Science 381:eadg7492, 2023. 10.1126/science.adg7492

Neither score is required: rarelens_ml.features treats a missing CADD or AlphaMissense value as NaN and LightGBM handles it, so the pipeline runs without the plugin data.

The simulated proband

make demo-case builds data/proband-simulated.vcf.gz: real GIAB HG002 variants as background plus one real ClinVar 2-star pathogenic variant in a disease gene (NF2 by default, giving neurofibromatosis type 2). Spiking a known variant into a public genome is how phenotype-driven triage tools are benchmarked, every input is public, and the file's header says SIMULATED. It is not a patient, and no part of it is invented: both the background and the planted variant are real published records.

The point of it is that the case has a right answer, so the ranking can be checked rather than admired. Give the case the phenotype of the planted disease and the planted variant should rank first — on phenotype, rarity and consequence, with ClinVar agreeing only afterwards.

Caveat when running without a VEP cache. VEP_DATABASE=true queries Ensembl's public database instead of the 25 GB cache. It returns no gnomAD frequencies, so every variant looks absent from gnomAD and the rarity term stops discriminating. Fine for showing the mechanics; use the cache for anything you would quote.

The model, and what its numbers mean

make training-set builds a training table straight from ClinVar rather than running VEP over hundreds of thousands of variants: ClinVar already carries the molecular consequence (MC), the gene (GENEINFO) and an allele frequency (AF_EXAC), which is the feature set serving sends. Only 2-star-and-above records are kept. make train then fits LightGBM and points the production alias at the new version.

The last run: 312,025 training and 74,239 held-out variants across 7,728 and 1,932 genes, with no gene on both sides.

	AUROC	AUPRC
all held-out variants	0.986	0.954
missense only (13,553)	0.872	0.725

Three things to say before anyone quotes the headline number:

1. 0.986 mostly measures how separable ClinVar's classes are by consequence. Its pathogenic set is largely loss of function and its benign set largely is not, so a model handed the consequence class does well without knowing anything hard. That is why the missense row exists: missense is where interpretation is actually difficult.
2. Even 0.872 is flattered by circularity. Within missense, every row has the same consequence and impact and no CADD or AlphaMissense score, so allele frequency is doing nearly all the work — and ClinVar's benign calls frequently use allele frequency as evidence (ACMG BA1/BS1). The feature partly caused the label.
3. It is not comparable to published CADD or AlphaMissense numbers. Those are trained and evaluated on different data. A fair comparison scores the same held-out rows with all three, which needs the plugin data (see above) and is the obvious next step.

Evaluating the model honestly

The model trains on ClinVar labels and is scored on ClinVar-labelled variants, which is exactly where published benchmarks go wrong. What to do about it:

1. Never let the label into the features. CLIN_SIG is excluded by construction; clinvar_sig is stored for display only (rarelens_ml/features.py lists the five feature columns).
2. Split by gene, not by variant. Random splits put variants from the same gene on both sides, and a model can then score a gene rather than a variant. Grimm et al. showed this inflates reported accuracy for exactly this class of tool: Hum Mutat 36:513–523, 2015. 10.1002/humu.22768 Implemented: rarelens_ml.train.split_by_gene holds out whole genes.
3. Prefer a time-based holdout. Train on an older ClinVar release (monthly archives live under vcf_GRCh38/archive_2.0/) and test only on variants classified after that date. This is the closest thing to a prospective evaluation available without new patients.
4. Filter labels by review status. ClinVar's CLNREVSTAT marks how much evidence backs a classification; two-star and above ("multiple submitters, no conflicts") is the usual bar. Known gap: VEP's CLIN_SIG does not carry review status, so this needs ClinVar annotated as a custom field before it can be enforced.
5. Report against published baselines on the same rows. CADD PHRED and AlphaMissense are already columns in the variant table, so AUROC and AUPRC for the model next to those two, with the variant count, is a fair comparison rather than a number with nothing to beat.
6. Report AUPRC, not just AUROC. Pathogenic variants are the minority class; AUROC flatters.

What must not be claimed

ACMG/AMP treats computational predictions as supporting evidence only, never sufficient on their own for classifying a variant (Richards et al., Genet Med 17:405–424, 2015. 10.1038/gim.2015.30). rarelens is a learning platform on public data: it makes no diagnostic claim, and the UI shows a score next to the evidence rather than a verdict. For what a real diagnostic pipeline looks like end to end, see the 100,000 Genomes Project rare-disease pilot: Smedley et al., N Engl J Med 385:1868–1880, 2021. 10.1056/NEJMoa2035790