Benchmarks
Two separate measurements: how well the phenotype ranking retrieves the right gene, and how well the model classifies pathogenicity. They are not the same thing and should never be quoted as one.
Phenotype ranking
make benchmark # downloads phenopacket-store, exports gene_phenotypes, runs the ranking
Given a real patient's reported HPO terms, where does the gene their authors diagnosed rank among all 5,269 HPO-annotated genes? Ties are reported as a range: optimistic counts a tie as a win, pessimistic counts every tied gene as ahead.
| cases | top-1 | top-10 | MRR | |
|---|---|---|---|---|
| all 10,178 | optimistic | 81.0% | 87.2% | 0.830 |
| pessimistic | 45.9% | 69.7% | 0.541 | |
| the 6,485 with ≥6 terms | optimistic | 77.1% | 85.4% | 0.797 |
| pessimistic | 59.5% | 81.0% | 0.670 |
Random guessing would be 0.02%.
The benchmark is contaminated
HPO's gene-to-phenotype annotations are curated from these same case reports. The median causal gene already carries every one of its patient's terms. So this measures how well the ranking retrieves a gene HPO has already been told about — an upper bound. A prospective number, on a patient whose gene nobody has annotated yet, would be lower, and this corpus cannot say by how much.
Do not quote the headline without this paragraph.
A change that did not work
Information-content weighting and ontology propagation both replaced plain term counting. Asked whether they helped — pessimistic figures, the 6,485 cases with six or more terms:
| scoring | top-1 | top-10 | MRR |
|---|---|---|---|
| count terms (the original) | 61.8% | 80.1% | 0.682 |
| + information content | 63.6% | 83.5% | 0.706 |
| + propagation | 58.2% | 77.7% | 0.653 |
| + both (shipped) | 59.5% | 81.0% | 0.670 |
Weighting earns its place. Propagation costs about what weighting gains. It was kept anyway, because this corpus cannot show what propagation is for — its term IDs were chosen by the same curators HPO records, so exact matching is flattered, while real users pick their own terms from a search box. That is an argument, not a measurement, and it is labelled as one.
The model
make training-set && make train
Held out by gene, never by variant: a random split puts variants of the same gene on both sides and the model scores the gene instead of the variant (Grimm et al., Hum Mutat 2015). 312,025 training and 74,239 test variants; 7,728 and 1,932 genes; no gene on both sides.
| AUROC | AUPRC | missense AUROC | missense AUPRC | |
|---|---|---|---|---|
| v2, with gnomAD allele frequency | 0.986 | 0.954 | 0.872 | 0.725 |
| v3, allele frequency removed | 0.966 | 0.881 | 0.500 | 0.398 |
0.500 is exactly random. Strip frequency out and nothing is left but the consequence class, so every missense variant scores identically. The respectable-looking 0.872 was allele frequency, not variant-effect knowledge — and ACMG's BA1/BS1 criteria assign ClinVar's benign labels using frequency, so the feature had partly caused the label.
The overall 0.966 is the same trick one level up: ClinVar's pathogenic set is largely loss of function and its benign set largely is not.
This is why the model abstains unless it has CADD or AlphaMissense. See Roadmap for why that is not yet a complete fix.
rarelens
Understanding it
Working on it
Running it
When it goes wrong