rarelens/docs/data.md

Data: what rarelens actually runs on

Everything below is public, peer-reviewed and consented for open redistribution. No patient data, no data access agreement, nothing that needs an application. These are the references to quote when showing the platform to someone.

Citations were verified against PubMed; each row links its DOI.

The demo slice

make data fetches two real files, chromosome 22 only (roughly 100 MB, minutes rather than hours):

File	What it is	Role
data/example.vcf.gz	GIAB HG002 (NA24385) v4.2.1 benchmark calls, GRCh38, chr22	the sample a scientist annotates
data/clinvar.chr22.vcf.gz	ClinVar, GRCh38, chr22	training labels, and the ClinVar column in the UI

HG002 is the NIST Genome in a Bottle Ashkenazi son, recruited through the Personal Genome Project, which consents participants to unrestricted public release. It is the reference genome the field benchmarks variant callers against, so it is both realistic and unambiguously shareable.

Datasets

Dataset	Used for	Access	Terms	Citation
ClinVar (GRCh38)	pathogenic/benign labels, ClinVar column	ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/	NCBI public domain	Landrum et al., Nucleic Acids Res 48(D1):D835–D844, 2020. 10.1093/nar/gkz972
Genome in a Bottle HG002 v4.2.1	the demo sample	ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/, s3://giab	open, no use restriction	Zook et al., Nat Biotechnol 37:561–566, 2019. 10.1038/s41587-019-0074-6
gnomAD v4	allele frequency feature and filter	gs://gcp-public-data--gnomad, s3://gnomad-public-us-east-1	free use, no restriction	Chen et al., Nature 625:92–100, 2024. 10.1038/s41586-023-06045-0; Karczewski et al., Nature 581:434–443, 2020. 10.1038/s41586-020-2308-7
1000 Genomes 30x	optional cohort/trio data	EBI FTP, s3://1000genomes	fully open, no access restriction	Byrska-Bishop et al., Cell 185(18):3426–3440.e19, 2022. 10.1016/j.cell.2022.08.004
MANE Select	one transcript per gene, if transcript choice ever matters	Ensembl/RefSeq	open	Morales et al., Nature 604:310–315, 2022. 10.1038/s41586-022-04558-8
Human Phenotype Ontology gene-to-phenotype	what the phenotype half of the ranking matches against (make hpo)	purl.obolibrary.org/obo/hp/hpoa/genes_to_phenotype.txt	free to use with attribution	Gargano et al., Nucleic Acids Res 52(D1):D1333–D1346, 2024. 10.1093/nar/gkad1005
Phenopacket Store	published patients: the reported phenotype and causal variant of a real case (make published-case)	github.com/monarch-initiative/phenopacket-store	BSD-3-Clause	Danis et al., HGG Adv 6(1):100371, 2025. 10.1016/j.xhgg.2024.100371

Tools and scores

Tool	Role	Terms	Citation
Ensembl VEP 113	annotation (pipeline/modules/vep.nf)	Apache 2.0	McLaren et al., Genome Biol 17:122, 2016. 10.1186/s13059-016-0974-4
CADD	cadd_phred feature	free for non-commercial use; commercial licence required	Rentzsch et al., Nucleic Acids Res 47(D1):D886–D894, 2019. 10.1093/nar/gky1016; Schubach et al., Nucleic Acids Res 52(D1), 2024. 10.1093/nar/gkad989
AlphaMissense	am_pathogenicity feature	predictions moved to CC BY 4.0 in March 2024 (originally CC BY-NC-SA)	Cheng et al., Science 381:eadg7492, 2023. 10.1126/science.adg7492

Neither score is required: rarelens_ml.features treats a missing CADD or AlphaMissense value as NaN and LightGBM handles it, so the pipeline runs without the plugin data.

The simulated proband

make demo-case builds data/proband-simulated.vcf.gz: real GIAB HG002 variants as background plus one real ClinVar 2-star pathogenic variant in a disease gene (NF2 by default, giving neurofibromatosis type 2). Spiking a known variant into a public genome is how phenotype-driven triage tools are benchmarked, every input is public, and the file's header says SIMULATED. It is not a patient, and no part of it is invented: both the background and the planted variant are real published records.

The point of it is that the case has a right answer, so the ranking can be checked rather than admired. Give the case the phenotype of the planted disease and the planted variant should rank first — on phenotype, rarity and consequence, with ClinVar agreeing only afterwards.

Caveat when running without a VEP cache. VEP_DATABASE=true queries Ensembl's public database instead of the 25 GB cache. It returns no gnomAD frequencies, so every variant looks absent from gnomAD and the rarity term stops discriminating. Fine for showing the mechanics; use the cache for anything you would quote.

A published case

make published-case builds a case around a patient who actually exists in the literature. It reads a GA4GH phenopacket from Monarch's Phenopacket Store, which curates published case reports into machine-readable records and keeps the PMID on each one, and takes two things from it verbatim: the phenotype terms the authors reported, and the variant they called causal.

The default is Loeys-Dietz syndrome, from the paper that first defined it — Loeys et al., Nat Genet 37:275–281, 2005, 10.1038/ng1511 (PMID 15731757). Family 4, individual II-1: 30 reported HPO terms, from hypertelorism and a bifid uvula to arterial tortuosity and an aortic root aneurysm, and a heterozygous TGFBR2 missense variant, NM_003242.6:c.1069G>T p.(Gly357Trp), at GRCh38 chr3:30672252 G>T.

The phenotype and the answer are real. The rest of that patient's genome is not public, and triage means nothing if the causal variant is the only variant in the file, so background variants come from GIAB HG002 in a 3 Mb window around the locus. The file is therefore a published diagnosis inside a public background genome — not anyone's exome. That is the standard construction for benchmarking phenotype-driven triage, and it is the reason this case can be redistributed at all.

The background is taken from coding exons wherever possible, using coding-exon coordinates from Ensembl's public REST API. This matters more than it sounds: of the ~4,000 HG002 variants in that window only 9 fall in coding exons, so a random sample is entirely intronic, the consequence filter throws all of it away, and the causal variant ends up the only candidate left — a funnel that proves nothing. Real coding variants give the ranking something it has to rank against.

What that run looks like: 21 variants in, 21 "rare" (see the caveat below), 2 surviving the consequence filter, 1 matching the phenotype.

	score	phenotype	rarity	consequence	model
TGFBR2 3:30672252 missense	0.897	1.00 (30/30 terms)	1.00	0.60	0.887
OSBPL10 3:31748090 missense	0.547	0.00	1.00	0.60	0.887

This is the whole argument for phenotype-driven triage in one table. Both are rare missense variants; the model scores them identically, to three decimal places, because nothing about the variants themselves distinguishes them. What separates the published diagnosis from an incidental variant in a lipid-transport gene is the patient's phenotype, and nothing else. ClinVar's "pathogenic" on the first row is shown afterwards as independent confirmation — it is not an input to the rank.

Three things to say out loud when showing it:

• The phenotype match is partly circular. HPO's gene-to-phenotype annotations are themselves curated from published cases, quite possibly including this one. A 30/30 term match against TGFBR2 is evidence the plumbing works, not evidence the ranking would find a novel gene.
• Rarity is not doing any work without a VEP cache. See the caveat above: in database mode every variant looks absent from gnomAD, so the funnel's rarity step passes everything and every variant scores a full 1.0 on rarity. --af_gnomade is rejected outright with --database, and plain --af returns nothing even for common variants — checked against rs429358, roughly 15% globally. Frequencies need the cache; there is no shortcut.
• The background is one healthy genome, not a diagnostic exome. A real case would have thousands of rare coding variants to discard, not a handful.

Other cases work the same way — any phenopacket with GRCh38 coordinates will do:

scripts/make-published-case.py --phenopacket <raw phenopacket-store JSON URL>

The model, and what its numbers mean

make training-set builds a training table straight from ClinVar rather than running VEP over hundreds of thousands of variants: ClinVar already carries the molecular consequence (MC), the gene (GENEINFO) and an allele frequency (AF_EXAC), which is the feature set serving sends. Only 2-star-and-above records are kept. make train then fits LightGBM and points the production alias at the new version.

The last run: 312,025 training and 74,239 held-out variants across 7,728 and 1,932 genes, with no gene on both sides.

	AUROC	AUPRC
all held-out variants	0.986	0.954
missense only (13,553)	0.872	0.725

Three things to say before anyone quotes the headline number:

1. 0.986 mostly measures how separable ClinVar's classes are by consequence. Its pathogenic set is largely loss of function and its benign set largely is not, so a model handed the consequence class does well without knowing anything hard. That is why the missense row exists: missense is where interpretation is actually difficult.
2. Even 0.872 is flattered by circularity. Within missense, every row has the same consequence and impact and no CADD or AlphaMissense score, so allele frequency is doing nearly all the work — and ClinVar's benign calls frequently use allele frequency as evidence (ACMG BA1/BS1). The feature partly caused the label.
3. It is not comparable to published CADD or AlphaMissense numbers. Those are trained and evaluated on different data. A fair comparison scores the same held-out rows with all three, which needs the plugin data (see above) and is the obvious next step.

Evaluating the model honestly

The model trains on ClinVar labels and is scored on ClinVar-labelled variants, which is exactly where published benchmarks go wrong. What to do about it:

1. Never let the label into the features. CLIN_SIG is excluded by construction; clinvar_sig is stored for display only (rarelens_ml/features.py lists the five feature columns).
2. Split by gene, not by variant. Random splits put variants from the same gene on both sides, and a model can then score a gene rather than a variant. Grimm et al. showed this inflates reported accuracy for exactly this class of tool: Hum Mutat 36:513–523, 2015. 10.1002/humu.22768 Implemented: rarelens_ml.train.split_by_gene holds out whole genes.
3. Prefer a time-based holdout. Train on an older ClinVar release (monthly archives live under vcf_GRCh38/archive_2.0/) and test only on variants classified after that date. This is the closest thing to a prospective evaluation available without new patients.
4. Filter labels by review status. ClinVar's CLNREVSTAT marks how much evidence backs a classification; two-star and above ("multiple submitters, no conflicts") is the usual bar. Known gap: VEP's CLIN_SIG does not carry review status, so this needs ClinVar annotated as a custom field before it can be enforced.
5. Report against published baselines on the same rows. CADD PHRED and AlphaMissense are already columns in the variant table, so AUROC and AUPRC for the model next to those two, with the variant count, is a fair comparison rather than a number with nothing to beat.
6. Report AUPRC, not just AUROC. Pathogenic variants are the minority class; AUROC flatters.

What must not be claimed

ACMG/AMP treats computational predictions as supporting evidence only, never sufficient on their own for classifying a variant (Richards et al., Genet Med 17:405–424, 2015. 10.1038/gim.2015.30). rarelens is a learning platform on public data: it makes no diagnostic claim, and the UI shows a score next to the evidence rather than a verdict. For what a real diagnostic pipeline looks like end to end, see the 100,000 Genomes Project rare-disease pilot: Smedley et al., N Engl J Med 385:1868–1880, 2021. 10.1056/NEJMoa2035790