kemal/rarelens
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix(science): stop scoring evidence that was never looked up

Browse Source 
A review of the ranking's arithmetic found four things wrong, all of which
made the score look better informed than it was. Measurements below are from
this repo, not estimates.

**Components now abstain instead of inventing a number.** A run without a VEP
cache returns no allele frequencies, and rarity_score(None) read that as
"absent from gnomAD, therefore maximally rare" and awarded every variant a
free 0.25. jobs.has_frequencies / has_effect_scores record what the run
actually produced, absent components are dropped from the weighted mean, and
the remaining weights are renormalised so the score keeps its meaning. The UI
shows "not looked up" rather than a bar, and the funnel stops calling a step
"rare" when nothing was filtered.

**Allele frequency is no longer a model feature.** It dominated: the same
missense variant scored 0.887 at AF 0 and 0.0003 at AF 0.01. That double-
counted, because the ranking already scores frequency explicitly, putting
~45% of every rank on one measurement; and it was circular, because ACMG
assigns ClinVar's benign labels using frequency (BA1/BS1). Retraining without
it moves missense AUROC from 0.872 to 0.500 — exactly random. The old figure
was allele frequency, not variant-effect knowledge. The model therefore
abstains unless CADD or AlphaMissense is present, since otherwise it only
restates the consequence class.

**Phenotype matching is weighted by information content** and HPO annotations
are propagated up the ontology. Counting terms alike let "global
developmental delay" (IC 0.93) count as much as "dilated left subclavian
artery" (IC 7.88).

**A real bug in the propagation, found by checking it.** The ancestor walk
read a pre-order DFS backwards, which on a DAG lets a term resolve before one
of its parents and inherit that parent alone instead of its lineage. It
dropped 399 terms out of the phenotype branch, Camptodactyly and Chiari
malformation among them. Now a true post-order, tested against a reference
transitive closure.

The ontology arithmetic moved to rarelens_ml.hpo so it is covered by tests,
and rarelens_ml.benchmark measures the whole thing: across 10,178 published
cases the causal gene ranks first 45.9-81.0% of the time against 5,269 genes,
versus 0.02% for chance. docs/data.md reports that with its contamination
(HPO's annotations come from these same case reports), and includes the
measurement showing information-content weighting earns its place while
propagation does not - kept anyway, for a reason the docs argue rather than
assume.
This commit is contained in:
Kemal Yaylali
2026-09-12 11:32:46 +01:00
parent 749b0f8214
commit e76ae847a1
37 changed files with 4324 additions and 195 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/data.md
+100 -42
View File
@@ -56,9 +56,11 @@ admired. Give the case the phenotype of the planted disease and the planted vari
first — on phenotype, rarity and consequence, with ClinVar agreeing only afterwards.
**Caveat when running without a VEP cache.** `VEP_DATABASE=true` queries Ensembl's public database
instead of the 25 GB cache. It returns no gnomAD frequencies, so every variant looks absent from
gnomAD and the rarity term stops discriminating. Fine for showing the mechanics; use the cache for
anything you would quote.
instead of the 25 GB cache. It returns no gnomAD frequencies and no plugin scores, so the rarity
and model components have nothing to work with. They abstain: the job records what the run looked
up (`jobs.has_frequencies`, `jobs.has_effect_scores`), those components are dropped from the score
rather than given a default, and the remaining weights are renormalised. Fine for showing the
mechanics; use the cache for anything you would quote.
## A published case
@@ -86,31 +88,31 @@ window only 9 fall in coding exons, so a random sample is entirely intronic, the
filter throws all of it away, and the causal variant ends up the only candidate left — a funnel
that proves nothing. Real coding variants give the ranking something it has to rank *against*.
What that run looks like: 21 variants in, 21 "rare" (see the caveat below), **2** surviving the
consequence filter, **1** matching the phenotype.
What that run looks like: 21 variants in, **2** surviving the consequence filter, **1** matching
the phenotype. Run without a VEP cache, so two of the four components have nothing to go on and
abstain; the remaining weights are renormalised over 0.55.
| | score | phenotype | rarity | consequence | model |
| | score | phenotype (0.64) | rarity | consequence (0.36) | model |
|---|---|---|---|---|---|
| *TGFBR2* 3:30672252 missense | **0.897** | 1.00 (30/30 terms) | 1.00 | 0.60 | 0.887 |
| *OSBPL10* 3:31748090 missense | 0.547 | 0.00 | 1.00 | 0.60 | 0.887 |
| *TGFBR2* 3:30672252 missense | **0.855** | 1.00 (30/30 terms) | not looked up | 0.60 | not looked up |
| *OSBPL10* 3:31748090 missense | 0.218 | 0.00 | not looked up | 0.60 | not looked up |
This is the whole argument for phenotype-driven triage in one table. Both are rare missense
variants; the model scores them **identically**, to three decimal places, because nothing about
the variants themselves distinguishes them. What separates the published diagnosis from an
incidental variant in a lipid-transport gene is the patient's phenotype, and nothing else.
ClinVar's "pathogenic" on the first row is shown afterwards as independent confirmation — it is
not an input to the rank.
variants, identical on every piece of evidence this run has except one. What separates the
published diagnosis from an incidental variant in a lipid-transport gene is the patient's
phenotype, and nothing else. ClinVar's "pathogenic" on the first row is shown afterwards as
independent confirmation — it is not an input to the rank.
Three things to say out loud when showing it:
- **The phenotype match is partly circular.** HPO's gene-to-phenotype annotations are themselves
curated from published cases, quite possibly including this one. A 30/30 term match against
*TGFBR2* is evidence the plumbing works, not evidence the ranking would find a novel gene.
- **Rarity is not doing any work without a VEP cache.** See the caveat above: in database mode
every variant looks absent from gnomAD, so the funnel's rarity step passes everything and every
variant scores a full 1.0 on rarity. `--af_gnomade` is rejected outright with `--database`, and
plain `--af` returns nothing even for common variants — checked against rs429358, roughly 15%
globally. Frequencies need the cache; there is no shortcut.
- **Two of the four components abstained, and that is the honest outcome.** In database mode VEP
returns no frequencies and no plugin scores, so rarity and the model have nothing to say.
`--af_gnomade` is rejected outright with `--database`, and plain `--af` returns nothing even for
common variants — checked against rs429358, roughly 15% globally. An earlier version of this
table read 1.00 for rarity on both rows, which was not a measurement: it was the absence of one.
- **The background is one healthy genome, not a diagnostic exome.** A real case would have
thousands of rare coding variants to discard, not a handful.
@@ -120,35 +122,90 @@ Other cases work the same way — any phenopacket with GRCh38 coordinates will d
scripts/make-published-case.py --phenopacket <raw phenopacket-store JSON URL>
```
## The model, and what its numbers mean
## The model, and why it currently abstains
`make training-set` builds a training table straight from ClinVar rather than running VEP over
hundreds of thousands of variants: ClinVar already carries the molecular consequence (`MC`), the
gene (`GENEINFO`) and an allele frequency (`AF_EXAC`), which is the feature set serving sends.
Only 2-star-and-above records are kept. `make train` then fits LightGBM and points the
`production` alias at the new version.
hundreds of thousands of variants: ClinVar already carries the molecular consequence (`MC`) and the
gene (`GENEINFO`). Only 2-star-and-above records are kept. `make train` then fits LightGBM and
points the `production` alias at the new version. 312,025 training and 74,239 held-out variants
across 7,728 and 1,932 genes, with no gene on both sides.
The last run: 312,025 training and 74,239 held-out variants across 7,728 and 1,932 genes, with no
gene on both sides.
The model used to take allele frequency as a feature. Removing it is the single most informative
thing in this document, because of what happened to the numbers:
| | AUROC | AUPRC |
|---|---|---|
| all held-out variants | 0.986 | 0.954 |
| missense only (13,553) | 0.872 | 0.725 |
| | AUROC | AUPRC | missense AUROC | missense AUPRC |
|---|---|---|---|---|
| v2, with gnomAD allele frequency | 0.986 | 0.954 | 0.872 | 0.725 |
| v3, allele frequency removed | 0.966 | 0.881 | **0.500** | 0.398 |
Three things to say before anyone quotes the headline number:
**0.500 on missense is exactly random.** Strip frequency out and the model cannot tell one missense
variant from another at all, because nothing is left but the consequence class — every missense row
gets the identical score. So the respectable-looking 0.872 was not variant-effect knowledge. It was
allele frequency, and ClinVar's benign calls are *made with* allele frequency under ACMG's BA1/BS1
criteria. The feature had partly caused the label; the model had rediscovered the labelling rule.
1. **0.986 mostly measures how separable ClinVar's classes are by consequence.** Its pathogenic set
is largely loss of function and its benign set largely is not, so a model handed the consequence
class does well without knowing anything hard. That is why the missense row exists: missense is
where interpretation is actually difficult.
2. **Even 0.872 is flattered by circularity.** Within missense, every row has the same consequence
and impact and no CADD or AlphaMissense score, so allele frequency is doing nearly all the work
— and ClinVar's benign calls frequently *use* allele frequency as evidence (ACMG BA1/BS1). The
feature partly caused the label.
3. **It is not comparable to published CADD or AlphaMissense numbers.** Those are trained and
evaluated on different data. A fair comparison scores the same held-out rows with all three,
which needs the plugin data (see above) and is the obvious next step.
The remaining 0.966 is the same trick one level up: ClinVar's pathogenic set is largely loss of
function and its benign set largely is not, so a model handed the consequence class separates them
without knowing anything hard.
Two consequences, both deliberate:
1. **Allele frequency is no longer a feature.** The ranking already scores frequency explicitly, as
a step function a reviewer can read (`triage.rarity_score`). Feeding it to the model as well put
roughly 45% of every rank on one measurement counted twice.
2. **The model abstains unless it has CADD or AlphaMissense.** Without them it can only restate the
consequence class, which the ranking already scores — and 0.500 is the measurement saying so.
Install the plugin data (see data/README.md) and the model earns its 0.20 back; until then it
contributes nothing, and the UI says so rather than showing a number.
It is also not comparable to published CADD or AlphaMissense figures, which are trained and
evaluated on different data. A fair comparison scores the same held-out rows with all three, which
needs the plugin data and is the obvious next step.
## How well does the phenotype ranking actually work?
`make benchmark` runs the ranking against every case in Phenopacket Store: given a real patient's
reported terms, where does the gene their authors diagnosed come in a ranking of all 5,269 genes
HPO annotates? Ties are reported as a range, because term-overlap scoring puts many genes on
identical scores — optimistic counts a tie as a win, pessimistic counts every tied gene as ahead.
| cases | | top-1 | top-10 | MRR |
|---|---|---|---|---|
| all 10,178 | optimistic | 81.0% | 87.2% | 0.830 |
| | pessimistic | 45.9% | 69.7% | 0.541 |
| the 6,485 with ≥6 terms | optimistic | 77.1% | 85.4% | 0.797 |
| | pessimistic | 59.5% | 81.0% | 0.670 |
Random guessing would put the right gene first 0.02% of the time, so the phenotype term is doing
real work. Two caveats, and the first is severe.
**The benchmark is contaminated.** The median causal gene already carries *every one* of its
patient's terms, because HPO's gene annotations are curated from these same case reports. This
measures how well the ranking retrieves a gene HPO has already been told about — an upper bound.
A prospective number, on a patient whose gene nobody has annotated yet, would be lower, and this
corpus cannot say by how much.
**Information content earns its place; propagation does not, measurably.** Both were added to
replace plain term counting, and the corpus was asked whether they helped. On the 6,485 cases with
at least six terms, pessimistic figures (the honest end of the range):
| scoring | top-1 | top-10 | MRR |
|---|---|---|---|
| direct annotations, count terms (the original) | 61.8% | 80.1% | 0.682 |
| direct annotations, weight by information content | **63.6%** | **83.5%** | **0.706** |
| propagated annotations, count terms | 58.2% | 77.7% | 0.653 |
| propagated + information content (shipped) | 59.5% | 81.0% | 0.670 |
Weighting by specificity helps: it breaks ties, which is exactly what it is for. Propagation costs
about as much as weighting gains, and the shipped combination is a wash against the original — a
point or two either way inside a contaminated benchmark.
Propagation is kept anyway, and the reason is worth stating plainly rather than hiding behind the
numbers. This corpus cannot show what propagation is for: its terms were chosen by the same
curators whose choices HPO records, so the IDs already line up and exact matching is flattered.
The app's users pick their own terms from a search box and will not line up that neatly. The
measurement is here so anyone who disagrees can act on it — the four rows above are one flag and
one argument to `make benchmark` apart.
## Evaluating the model honestly
@@ -156,7 +213,8 @@ The model trains on ClinVar labels and is scored on ClinVar-labelled variants, w
where published benchmarks go wrong. What to do about it:
1. **Never let the label into the features.** `CLIN_SIG` is excluded by construction; `clinvar_sig`
is stored for display only (`rarelens_ml/features.py` lists the five feature columns).
is stored for display only (`rarelens_ml/features.py` lists the feature columns). Allele
frequency was removed for a related reason: ACMG uses it to *assign* the benign label.
2. **Split by gene, not by variant.** Random splits put variants from the same gene on both sides,
and a model can then score a gene rather than a variant. Grimm et al. showed this inflates
reported accuracy for exactly this class of tool: *Hum Mutat* 36:513–523, 2015.