fix(science): stop scoring evidence that was never looked up

A review of the ranking's arithmetic found four things wrong, all of which
made the score look better informed than it was. Measurements below are from
this repo, not estimates.

**Components now abstain instead of inventing a number.** A run without a VEP
cache returns no allele frequencies, and rarity_score(None) read that as
"absent from gnomAD, therefore maximally rare" and awarded every variant a
free 0.25. jobs.has_frequencies / has_effect_scores record what the run
actually produced, absent components are dropped from the weighted mean, and
the remaining weights are renormalised so the score keeps its meaning. The UI
shows "not looked up" rather than a bar, and the funnel stops calling a step
"rare" when nothing was filtered.

**Allele frequency is no longer a model feature.** It dominated: the same
missense variant scored 0.887 at AF 0 and 0.0003 at AF 0.01. That double-
counted, because the ranking already scores frequency explicitly, putting
~45% of every rank on one measurement; and it was circular, because ACMG
assigns ClinVar's benign labels using frequency (BA1/BS1). Retraining without
it moves missense AUROC from 0.872 to 0.500 — exactly random. The old figure
was allele frequency, not variant-effect knowledge. The model therefore
abstains unless CADD or AlphaMissense is present, since otherwise it only
restates the consequence class.

**Phenotype matching is weighted by information content** and HPO annotations
are propagated up the ontology. Counting terms alike let "global
developmental delay" (IC 0.93) count as much as "dilated left subclavian
artery" (IC 7.88).

**A real bug in the propagation, found by checking it.** The ancestor walk
read a pre-order DFS backwards, which on a DAG lets a term resolve before one
of its parents and inherit that parent alone instead of its lineage. It
dropped 399 terms out of the phenotype branch, Camptodactyly and Chiari
malformation among them. Now a true post-order, tested against a reference
transitive closure.

The ontology arithmetic moved to rarelens_ml.hpo so it is covered by tests,
and rarelens_ml.benchmark measures the whole thing: across 10,178 published
cases the causal gene ranks first 45.9-81.0% of the time against 5,269 genes,
versus 0.02% for chance. docs/data.md reports that with its contamination
(HPO's annotations come from these same case reports), and includes the
measurement showing information-content weighting earns its place while
propagation does not - kept anyway, for a reason the docs argue rather than
assume.
This commit is contained in:
Kemal Yaylali
2026-09-12 11:32:46 +01:00
parent 749b0f8214
commit e76ae847a1
37 changed files with 4324 additions and 195 deletions
+16 -3
View File
@@ -1,7 +1,9 @@
.PHONY: up down clean migrate test lint data hpo demo-case published-case training-set train loader pipeline annotate images kind serverless-deploy serverless-destroy gcp-configure gcp-secrets
.PHONY: up down clean migrate test lint data hpo demo-case published-case benchmark training-set train loader pipeline annotate images kind serverless-deploy serverless-destroy gcp-configure gcp-secrets
VCF ?= data/example.vcf.gz
MLFLOW_URI ?= http://localhost:5001
DB_CONTAINER ?= rarelens-db-1
BENCH_DIR ?= data
TAG ?= latest
# The loader container reaches docker-compose's Postgres through the host.
HOST_DB_URL ?= postgresql://rarelens:[email protected]:5432/rarelens
@@ -31,8 +33,8 @@ lint:
data: ## download the public demo slice: GIAB HG002 + ClinVar, chr22 (see docs/data.md)
scripts/fetch-demo-data.sh
hpo: ## load HPO gene-to-phenotype annotations, which the ranking matches against
scripts/load-hpo.py
hpo: ## load HPO annotations, propagated up the ontology and weighted by information content
cd ml && uv run --extra db python ../scripts/load-hpo.py
demo-case: ## build the simulated proband: GIAB background + one ClinVar pathogenic variant
scripts/make-demo-case.sh
@@ -43,6 +45,17 @@ published-case: ## build a case from a published patient: a GA4GH phenopacket +
training-set: ## build a ClinVar training table, shaped like VEP --tab output
scripts/make-training-set.sh
benchmark: ## measure the phenotype ranking against every published case (see docs/data.md)
docker exec $(DB_CONTAINER) psql -U rarelens -d rarelens -At -F',' \
-c "select gene_symbol, hpo_id from gene_phenotypes" \
| tr ',' '\t' > $(BENCH_DIR)/gene_phenotypes.tsv
test -f $(BENCH_DIR)/all_phenopackets.zip || curl -sL -o $(BENCH_DIR)/all_phenopackets.zip \
"$$(curl -s https://api.github.com/repos/monarch-initiative/phenopacket-store/releases/latest \
| sed -n 's/.*"browser_download_url": "\(.*all_phenopackets.zip\)".*/\1/p')"
cd ml && uv run --extra dev python -m rarelens_ml.benchmark \
--phenopackets ../$(BENCH_DIR)/all_phenopackets.zip \
--annotations ../$(BENCH_DIR)/gene_phenotypes.tsv
train: ## train the pathogenicity model and point the production alias at it (needs `make up`)
cd ml && MLFLOW_TRACKING_URI=$(MLFLOW_URI) uv run --extra dev \
python -m rarelens_ml.train --tsv ../data/clinvar-training.vep.tsv --register