feat(ml): train a real model, and report the number that matters rather than the flattering one

"Variants are unscored" was accurate: nothing was ever trained, so a quarter of every rank was
dead weight and the UI leaked a connection error at the reader.

- scripts/make-training-set.sh derives a training table from ClinVar directly. ClinVar already
  carries the molecular consequence, the gene and an allele frequency, which is the feature set
  serving sends, so this avoids running VEP over hundreds of thousands of variants. 2-star
  records only.
- train.py now holds out whole genes (GroupShuffleSplit). docs/data.md had said to do this since
  the data pass; the code was still doing a random split, which is the leak Grimm 2015 describes.
- evaluate() reports missense on its own. On the last run: AUROC 0.986 over 74,239 held-out
  variants, but 0.872 over the 13,553 missense ones, and the docs say plainly why even that is
  flattered — within missense the only live feature is allele frequency, and ClinVar's benign
  calls often use allele frequency as evidence (ACMG BA1/BS1), so the feature partly caused the
  label.
- the 503 now names what is missing (model@alias via tracking URI) and leaves the exception in
  the server log instead of the UI.
- make training-set / make train; the 58 MB table is gitignored.

Verified end to end: model registered as v2, the simulated NF2 case scores 0.999 on the planted
variant, and it now ranks 1.00 with all four components live.

Tests: api 77, ml 22, loader 16, web 32; ruff, mypy, svelte-check clean.
This commit is contained in:
Kemal Yaylali
2026-09-12 09:13:54 +01:00
parent 3ab404ebe5
commit 197975cc42
9 changed files with 235 additions and 14 deletions
+32 -1
View File
@@ -59,6 +59,36 @@ instead of the 25 GB cache. It returns no gnomAD frequencies, so every variant l
gnomAD and the rarity term stops discriminating. Fine for showing the mechanics; use the cache for
anything you would quote.
## The model, and what its numbers mean
`make training-set` builds a training table straight from ClinVar rather than running VEP over
hundreds of thousands of variants: ClinVar already carries the molecular consequence (`MC`), the
gene (`GENEINFO`) and an allele frequency (`AF_EXAC`), which is the feature set serving sends.
Only 2-star-and-above records are kept. `make train` then fits LightGBM and points the
`production` alias at the new version.
The last run: 312,025 training and 74,239 held-out variants across 7,728 and 1,932 genes, with no
gene on both sides.
| | AUROC | AUPRC |
|---|---|---|
| all held-out variants | 0.986 | 0.954 |
| missense only (13,553) | 0.872 | 0.725 |
Three things to say before anyone quotes the headline number:
1. **0.986 mostly measures how separable ClinVar's classes are by consequence.** Its pathogenic set
is largely loss of function and its benign set largely is not, so a model handed the consequence
class does well without knowing anything hard. That is why the missense row exists: missense is
where interpretation is actually difficult.
2. **Even 0.872 is flattered by circularity.** Within missense, every row has the same consequence
and impact and no CADD or AlphaMissense score, so allele frequency is doing nearly all the work
— and ClinVar's benign calls frequently *use* allele frequency as evidence (ACMG BA1/BS1). The
feature partly caused the label.
3. **It is not comparable to published CADD or AlphaMissense numbers.** Those are trained and
evaluated on different data. A fair comparison scores the same held-out rows with all three,
which needs the plugin data (see above) and is the obvious next step.
## Evaluating the model honestly
The model trains on ClinVar labels and is scored on ClinVar-labelled variants, which is exactly
@@ -69,7 +99,8 @@ where published benchmarks go wrong. What to do about it:
2. **Split by gene, not by variant.** Random splits put variants from the same gene on both sides,
and a model can then score a gene rather than a variant. Grimm et al. showed this inflates
reported accuracy for exactly this class of tool: *Hum Mutat* 36:513523, 2015.
[10.1002/humu.22768](https://doi.org/10.1002/humu.22768)
[10.1002/humu.22768](https://doi.org/10.1002/humu.22768) *Implemented*:
`rarelens_ml.train.split_by_gene` holds out whole genes.
3. **Prefer a time-based holdout.** Train on an older ClinVar release (monthly archives live under
`vcf_GRCh38/archive_2.0/`) and test only on variants classified after that date. This is the
closest thing to a prospective evaluation available without new patients.