Kemal Yaylali
197975cc42
"Variants are unscored" was accurate: nothing was ever trained, so a quarter of every rank was dead weight and the UI leaked a connection error at the reader. - scripts/make-training-set.sh derives a training table from ClinVar directly. ClinVar already carries the molecular consequence, the gene and an allele frequency, which is the feature set serving sends, so this avoids running VEP over hundreds of thousands of variants. 2-star records only. - train.py now holds out whole genes (GroupShuffleSplit). docs/data.md had said to do this since the data pass; the code was still doing a random split, which is the leak Grimm 2015 describes. - evaluate() reports missense on its own. On the last run: AUROC 0.986 over 74,239 held-out variants, but 0.872 over the 13,553 missense ones, and the docs say plainly why even that is flattered — within missense the only live feature is allele frequency, and ClinVar's benign calls often use allele frequency as evidence (ACMG BA1/BS1), so the feature partly caused the label. - the 503 now names what is missing (model@alias via tracking URI) and leaves the exception in the server log instead of the UI. - make training-set / make train; the 58 MB table is gitignored. Verified end to end: model registered as v2, the simulated NF2 case scores 0.999 on the planted variant, and it now ranks 1.00 with all four components live. Tests: api 77, ml 22, loader 16, web 32; ruff, mypy, svelte-check clean.
125 lines
8.9 KiB
Markdown
125 lines
8.9 KiB
Markdown
# Data: what rarelens actually runs on
|
||
|
||
Everything below is public, peer-reviewed and consented for open redistribution. No patient data,
|
||
no data access agreement, nothing that needs an application. These are the references to quote
|
||
when showing the platform to someone.
|
||
|
||
Citations were verified against [PubMed](https://pubmed.ncbi.nlm.nih.gov/); each row links its DOI.
|
||
|
||
## The demo slice
|
||
|
||
`make data` fetches two real files, chromosome 22 only (roughly 100 MB, minutes rather than hours):
|
||
|
||
| File | What it is | Role |
|
||
|---|---|---|
|
||
| `data/example.vcf.gz` | GIAB HG002 (NA24385) v4.2.1 benchmark calls, GRCh38, chr22 | the sample a scientist annotates |
|
||
| `data/clinvar.chr22.vcf.gz` | ClinVar, GRCh38, chr22 | training labels, and the ClinVar column in the UI |
|
||
|
||
HG002 is the NIST Genome in a Bottle Ashkenazi son, recruited through the Personal Genome Project,
|
||
which consents participants to unrestricted public release. It is the reference genome the field
|
||
benchmarks variant callers against, so it is both realistic and unambiguously shareable.
|
||
|
||
## Datasets
|
||
|
||
| Dataset | Used for | Access | Terms | Citation |
|
||
|---|---|---|---|---|
|
||
| **ClinVar** (GRCh38) | pathogenic/benign labels, ClinVar column | `ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/` | NCBI public domain | Landrum et al., *Nucleic Acids Res* 48(D1):D835–D844, 2020. [10.1093/nar/gkz972](https://doi.org/10.1093/nar/gkz972) |
|
||
| **Genome in a Bottle** HG002 v4.2.1 | the demo sample | `ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/`, `s3://giab` | open, no use restriction | Zook et al., *Nat Biotechnol* 37:561–566, 2019. [10.1038/s41587-019-0074-6](https://doi.org/10.1038/s41587-019-0074-6) |
|
||
| **gnomAD** v4 | allele frequency feature and filter | `gs://gcp-public-data--gnomad`, `s3://gnomad-public-us-east-1` | free use, no restriction | Chen et al., *Nature* 625:92–100, 2024. [10.1038/s41586-023-06045-0](https://doi.org/10.1038/s41586-023-06045-0); Karczewski et al., *Nature* 581:434–443, 2020. [10.1038/s41586-020-2308-7](https://doi.org/10.1038/s41586-020-2308-7) |
|
||
| **1000 Genomes** 30x | optional cohort/trio data | EBI FTP, `s3://1000genomes` | fully open, no access restriction | Byrska-Bishop et al., *Cell* 185(18):3426–3440.e19, 2022. [10.1016/j.cell.2022.08.004](https://doi.org/10.1016/j.cell.2022.08.004) |
|
||
| **MANE Select** | one transcript per gene, if transcript choice ever matters | Ensembl/RefSeq | open | Morales et al., *Nature* 604:310–315, 2022. [10.1038/s41586-022-04558-8](https://doi.org/10.1038/s41586-022-04558-8) |
|
||
| **Human Phenotype Ontology** gene-to-phenotype | what the phenotype half of the ranking matches against (`make hpo`) | `purl.obolibrary.org/obo/hp/hpoa/genes_to_phenotype.txt` | free to use with attribution | Gargano et al., *Nucleic Acids Res* 52(D1):D1333–D1346, 2024. [10.1093/nar/gkad1005](https://doi.org/10.1093/nar/gkad1005) |
|
||
|
||
## Tools and scores
|
||
|
||
| Tool | Role | Terms | Citation |
|
||
|---|---|---|---|
|
||
| **Ensembl VEP** 113 | annotation (`pipeline/modules/vep.nf`) | Apache 2.0 | McLaren et al., *Genome Biol* 17:122, 2016. [10.1186/s13059-016-0974-4](https://doi.org/10.1186/s13059-016-0974-4) |
|
||
| **CADD** | `cadd_phred` feature | free for non-commercial use; commercial licence required | Rentzsch et al., *Nucleic Acids Res* 47(D1):D886–D894, 2019. [10.1093/nar/gky1016](https://doi.org/10.1093/nar/gky1016); Schubach et al., *Nucleic Acids Res* 52(D1), 2024. [10.1093/nar/gkad989](https://doi.org/10.1093/nar/gkad989) |
|
||
| **AlphaMissense** | `am_pathogenicity` feature | predictions moved to CC BY 4.0 in March 2024 (originally CC BY-NC-SA) | Cheng et al., *Science* 381:eadg7492, 2023. [10.1126/science.adg7492](https://doi.org/10.1126/science.adg7492) |
|
||
|
||
Neither score is required: `rarelens_ml.features` treats a missing CADD or AlphaMissense value as
|
||
NaN and LightGBM handles it, so the pipeline runs without the plugin data.
|
||
|
||
## The simulated proband
|
||
|
||
`make demo-case` builds `data/proband-simulated.vcf.gz`: real GIAB HG002 variants as background
|
||
plus one real ClinVar 2-star pathogenic variant in a disease gene (NF2 by default, giving
|
||
neurofibromatosis type 2). Spiking a known variant into a public genome is how phenotype-driven
|
||
triage tools are benchmarked, every input is public, and the file's header says SIMULATED. It is
|
||
not a patient, and no part of it is invented: both the background and the planted variant are real
|
||
published records.
|
||
|
||
The point of it is that the case has a right answer, so the ranking can be checked rather than
|
||
admired. Give the case the phenotype of the planted disease and the planted variant should rank
|
||
first — on phenotype, rarity and consequence, with ClinVar agreeing only afterwards.
|
||
|
||
**Caveat when running without a VEP cache.** `VEP_DATABASE=true` queries Ensembl's public database
|
||
instead of the 25 GB cache. It returns no gnomAD frequencies, so every variant looks absent from
|
||
gnomAD and the rarity term stops discriminating. Fine for showing the mechanics; use the cache for
|
||
anything you would quote.
|
||
|
||
## The model, and what its numbers mean
|
||
|
||
`make training-set` builds a training table straight from ClinVar rather than running VEP over
|
||
hundreds of thousands of variants: ClinVar already carries the molecular consequence (`MC`), the
|
||
gene (`GENEINFO`) and an allele frequency (`AF_EXAC`), which is the feature set serving sends.
|
||
Only 2-star-and-above records are kept. `make train` then fits LightGBM and points the
|
||
`production` alias at the new version.
|
||
|
||
The last run: 312,025 training and 74,239 held-out variants across 7,728 and 1,932 genes, with no
|
||
gene on both sides.
|
||
|
||
| | AUROC | AUPRC |
|
||
|---|---|---|
|
||
| all held-out variants | 0.986 | 0.954 |
|
||
| missense only (13,553) | 0.872 | 0.725 |
|
||
|
||
Three things to say before anyone quotes the headline number:
|
||
|
||
1. **0.986 mostly measures how separable ClinVar's classes are by consequence.** Its pathogenic set
|
||
is largely loss of function and its benign set largely is not, so a model handed the consequence
|
||
class does well without knowing anything hard. That is why the missense row exists: missense is
|
||
where interpretation is actually difficult.
|
||
2. **Even 0.872 is flattered by circularity.** Within missense, every row has the same consequence
|
||
and impact and no CADD or AlphaMissense score, so allele frequency is doing nearly all the work
|
||
— and ClinVar's benign calls frequently *use* allele frequency as evidence (ACMG BA1/BS1). The
|
||
feature partly caused the label.
|
||
3. **It is not comparable to published CADD or AlphaMissense numbers.** Those are trained and
|
||
evaluated on different data. A fair comparison scores the same held-out rows with all three,
|
||
which needs the plugin data (see above) and is the obvious next step.
|
||
|
||
## Evaluating the model honestly
|
||
|
||
The model trains on ClinVar labels and is scored on ClinVar-labelled variants, which is exactly
|
||
where published benchmarks go wrong. What to do about it:
|
||
|
||
1. **Never let the label into the features.** `CLIN_SIG` is excluded by construction; `clinvar_sig`
|
||
is stored for display only (`rarelens_ml/features.py` lists the five feature columns).
|
||
2. **Split by gene, not by variant.** Random splits put variants from the same gene on both sides,
|
||
and a model can then score a gene rather than a variant. Grimm et al. showed this inflates
|
||
reported accuracy for exactly this class of tool: *Hum Mutat* 36:513–523, 2015.
|
||
[10.1002/humu.22768](https://doi.org/10.1002/humu.22768) *Implemented*:
|
||
`rarelens_ml.train.split_by_gene` holds out whole genes.
|
||
3. **Prefer a time-based holdout.** Train on an older ClinVar release (monthly archives live under
|
||
`vcf_GRCh38/archive_2.0/`) and test only on variants classified after that date. This is the
|
||
closest thing to a prospective evaluation available without new patients.
|
||
4. **Filter labels by review status.** ClinVar's `CLNREVSTAT` marks how much evidence backs a
|
||
classification; two-star and above ("multiple submitters, no conflicts") is the usual bar.
|
||
*Known gap*: VEP's `CLIN_SIG` does not carry review status, so this needs ClinVar annotated as a
|
||
custom field before it can be enforced.
|
||
5. **Report against published baselines on the same rows.** CADD PHRED and AlphaMissense are
|
||
already columns in the variant table, so AUROC and AUPRC for the model next to those two, with
|
||
the variant count, is a fair comparison rather than a number with nothing to beat.
|
||
6. **Report AUPRC, not just AUROC.** Pathogenic variants are the minority class; AUROC flatters.
|
||
|
||
## What must not be claimed
|
||
|
||
ACMG/AMP treats computational predictions as *supporting* evidence only, never sufficient on their
|
||
own for classifying a variant (Richards et al., *Genet Med* 17:405–424, 2015.
|
||
[10.1038/gim.2015.30](https://doi.org/10.1038/gim.2015.30)). rarelens is a learning platform on
|
||
public data: it makes no diagnostic claim, and the UI shows a score next to the evidence rather
|
||
than a verdict. For what a real diagnostic pipeline looks like end to end, see the 100,000 Genomes
|
||
Project rare-disease pilot: Smedley et al., *N Engl J Med* 385:1868–1880, 2021.
|
||
[10.1056/NEJMoa2035790](https://doi.org/10.1056/NEJMoa2035790)
|