# Data: what rarelens actually runs on

Everything below is public, peer-reviewed and consented for open redistribution. No patient data,
no data access agreement, nothing that needs an application. These are the references to quote
when showing the platform to someone.

Citations were verified against [PubMed](https://pubmed.ncbi.nlm.nih.gov/); each row links its DOI.

## The demo slice

`make data` fetches two real files, chromosome 22 only (roughly 100 MB, minutes rather than hours):

| File | What it is | Role |
|---|---|---|
| `data/example.vcf.gz` | GIAB HG002 (NA24385) v4.2.1 benchmark calls, GRCh38, chr22 | the sample a scientist annotates |
| `data/clinvar.chr22.vcf.gz` | ClinVar, GRCh38, chr22 | training labels, and the ClinVar column in the UI |

HG002 is the NIST Genome in a Bottle Ashkenazi son, recruited through the Personal Genome Project,
which consents participants to unrestricted public release. It is the reference genome the field
benchmarks variant callers against, so it is both realistic and unambiguously shareable.

## Datasets

| Dataset | Used for | Access | Terms | Citation |
|---|---|---|---|---|
| **ClinVar** (GRCh38) | pathogenic/benign labels, ClinVar column | `ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/` | NCBI public domain | Landrum et al., *Nucleic Acids Res* 48(D1):D835–D844, 2020. [10.1093/nar/gkz972](https://doi.org/10.1093/nar/gkz972) |
| **Genome in a Bottle** HG002 v4.2.1 | the demo sample | `ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/`, `s3://giab` | open, no use restriction | Zook et al., *Nat Biotechnol* 37:561–566, 2019. [10.1038/s41587-019-0074-6](https://doi.org/10.1038/s41587-019-0074-6) |
| **gnomAD** v4 | allele frequency feature and filter | `gs://gcp-public-data--gnomad`, `s3://gnomad-public-us-east-1` | free use, no restriction | Chen et al., *Nature* 625:92–100, 2024. [10.1038/s41586-023-06045-0](https://doi.org/10.1038/s41586-023-06045-0); Karczewski et al., *Nature* 581:434–443, 2020. [10.1038/s41586-020-2308-7](https://doi.org/10.1038/s41586-020-2308-7) |
| **1000 Genomes** 30x | optional cohort/trio data | EBI FTP, `s3://1000genomes` | fully open, no access restriction | Byrska-Bishop et al., *Cell* 185(18):3426–3440.e19, 2022. [10.1016/j.cell.2022.08.004](https://doi.org/10.1016/j.cell.2022.08.004) |
| **MANE Select** | one transcript per gene, if transcript choice ever matters | Ensembl/RefSeq | open | Morales et al., *Nature* 604:310–315, 2022. [10.1038/s41586-022-04558-8](https://doi.org/10.1038/s41586-022-04558-8) |

## Tools and scores

| Tool | Role | Terms | Citation |
|---|---|---|---|
| **Ensembl VEP** 113 | annotation (`pipeline/modules/vep.nf`) | Apache 2.0 | McLaren et al., *Genome Biol* 17:122, 2016. [10.1186/s13059-016-0974-4](https://doi.org/10.1186/s13059-016-0974-4) |
| **CADD** | `cadd_phred` feature | free for non-commercial use; commercial licence required | Rentzsch et al., *Nucleic Acids Res* 47(D1):D886–D894, 2019. [10.1093/nar/gky1016](https://doi.org/10.1093/nar/gky1016); Schubach et al., *Nucleic Acids Res* 52(D1), 2024. [10.1093/nar/gkad989](https://doi.org/10.1093/nar/gkad989) |
| **AlphaMissense** | `am_pathogenicity` feature | predictions moved to CC BY 4.0 in March 2024 (originally CC BY-NC-SA) | Cheng et al., *Science* 381:eadg7492, 2023. [10.1126/science.adg7492](https://doi.org/10.1126/science.adg7492) |

Neither score is required: `rarelens_ml.features` treats a missing CADD or AlphaMissense value as
NaN and LightGBM handles it, so the pipeline runs without the plugin data.

## Evaluating the model honestly

The model trains on ClinVar labels and is scored on ClinVar-labelled variants, which is exactly
where published benchmarks go wrong. What to do about it:

1. **Never let the label into the features.** `CLIN_SIG` is excluded by construction; `clinvar_sig`
   is stored for display only (`rarelens_ml/features.py` lists the five feature columns).
2. **Split by gene, not by variant.** Random splits put variants from the same gene on both sides,
   and a model can then score a gene rather than a variant. Grimm et al. showed this inflates
   reported accuracy for exactly this class of tool: *Hum Mutat* 36:513–523, 2015.
   [10.1002/humu.22768](https://doi.org/10.1002/humu.22768)
3. **Prefer a time-based holdout.** Train on an older ClinVar release (monthly archives live under
   `vcf_GRCh38/archive_2.0/`) and test only on variants classified after that date. This is the
   closest thing to a prospective evaluation available without new patients.
4. **Filter labels by review status.** ClinVar's `CLNREVSTAT` marks how much evidence backs a
   classification; two-star and above ("multiple submitters, no conflicts") is the usual bar.
   *Known gap*: VEP's `CLIN_SIG` does not carry review status, so this needs ClinVar annotated as a
   custom field before it can be enforced.
5. **Report against published baselines on the same rows.** CADD PHRED and AlphaMissense are
   already columns in the variant table, so AUROC and AUPRC for the model next to those two, with
   the variant count, is a fair comparison rather than a number with nothing to beat.
6. **Report AUPRC, not just AUROC.** Pathogenic variants are the minority class; AUROC flatters.

## What must not be claimed

ACMG/AMP treats computational predictions as *supporting* evidence only, never sufficient on their
own for classifying a variant (Richards et al., *Genet Med* 17:405–424, 2015.
[10.1038/gim.2015.30](https://doi.org/10.1038/gim.2015.30)). rarelens is a learning platform on
public data: it makes no diagnostic claim, and the UI shows a score next to the evidence rather
than a verdict. For what a real diagnostic pipeline looks like end to end, see the 100,000 Genomes
Project rare-disease pilot: Smedley et al., *N Engl J Med* 385:1868–1880, 2021.
[10.1056/NEJMoa2035790](https://doi.org/10.1056/NEJMoa2035790)