docs/blog/rarelens.tex builds a 10-page write-up covering the Nextflow DSL2
pipeline, the three execution backends behind one API call, the Argo and
ArgoCD track, Terraform, the external systems integrated, the ranking, the
benchmark and the model.
It is deliberately as much an account of what was wrong as of what works: the
components that scored evidence nobody had looked up, the missense AUROC that
fell from 0.872 to 0.500 once allele frequency was removed, the ontology walk
that silently dropped 399 terms, and the propagation change that measured
slightly worse than what it replaced and was kept anyway with the numbers
published.
Figures come from one headless-browser script. Screenshots are of the running
application; the six diagrams are hand-written HTML styled from the same
palette, rendered by the same script. Keeping both in one place is what stops
the article drifting from the system, and every number in a diagram has to be
changed deliberately.
A review of the ranking's arithmetic found four things wrong, all of which
made the score look better informed than it was. Measurements below are from
this repo, not estimates.
**Components now abstain instead of inventing a number.** A run without a VEP
cache returns no allele frequencies, and rarity_score(None) read that as
"absent from gnomAD, therefore maximally rare" and awarded every variant a
free 0.25. jobs.has_frequencies / has_effect_scores record what the run
actually produced, absent components are dropped from the weighted mean, and
the remaining weights are renormalised so the score keeps its meaning. The UI
shows "not looked up" rather than a bar, and the funnel stops calling a step
"rare" when nothing was filtered.
**Allele frequency is no longer a model feature.** It dominated: the same
missense variant scored 0.887 at AF 0 and 0.0003 at AF 0.01. That double-
counted, because the ranking already scores frequency explicitly, putting
~45% of every rank on one measurement; and it was circular, because ACMG
assigns ClinVar's benign labels using frequency (BA1/BS1). Retraining without
it moves missense AUROC from 0.872 to 0.500 — exactly random. The old figure
was allele frequency, not variant-effect knowledge. The model therefore
abstains unless CADD or AlphaMissense is present, since otherwise it only
restates the consequence class.
**Phenotype matching is weighted by information content** and HPO annotations
are propagated up the ontology. Counting terms alike let "global
developmental delay" (IC 0.93) count as much as "dilated left subclavian
artery" (IC 7.88).
**A real bug in the propagation, found by checking it.** The ancestor walk
read a pre-order DFS backwards, which on a DAG lets a term resolve before one
of its parents and inherit that parent alone instead of its lineage. It
dropped 399 terms out of the phenotype branch, Camptodactyly and Chiari
malformation among them. Now a true post-order, tested against a reference
transitive closure.
The ontology arithmetic moved to rarelens_ml.hpo so it is covered by tests,
and rarelens_ml.benchmark measures the whole thing: across 10,178 published
cases the causal gene ranks first 45.9-81.0% of the time against 5,269 genes,
versus 0.02% for chance. docs/data.md reports that with its contamination
(HPO's annotations come from these same case reports), and includes the
measurement showing information-content weighting earns its place while
propagation does not - kept anyway, for a reason the docs argue rather than
assume.
`make published-case` reads a GA4GH phenopacket from Monarch's Phenopacket
Store and takes two things from it verbatim: the HPO terms the authors
reported and the variant they called causal. The default is the TGFBR2
proband from Loeys et al., Nat Genet 2005 (doi:10.1038/ng1511), the paper
that first defined Loeys-Dietz syndrome -- 30 reported terms and
NM_003242.6:c.1069G>T p.(Gly357Trp).
The rest of that patient's genome is not public, so background variants come
from GIAB HG002 around the locus. They are drawn from coding exons where
possible, via Ensembl's REST API: of ~4,000 HG002 variants in the window only
9 are coding, so a random sample is entirely intronic, the consequence filter
discards all of it, and the causal variant is left as the only candidate --
a funnel that proves nothing.
The real run ranks TGFBR2 first at 0.897 against an OSBPL10 missense at
0.547. Both are rare missense variants the model scores identically (0.887);
only the phenotype separates them, which is the argument for phenotype-driven
triage in one table.
Documented with three caveats rather than left implicit: the phenotype match
is partly circular because HPO's gene annotations are themselves curated from
published cases; rarity contributes nothing without the VEP cache
(--af_gnomade is rejected with --database, and plain --af returns nothing
even for rs429358 at ~15% global frequency); and one healthy genome is not a
diagnostic exome.
"Variants are unscored" was accurate: nothing was ever trained, so a quarter of every rank was
dead weight and the UI leaked a connection error at the reader.
- scripts/make-training-set.sh derives a training table from ClinVar directly. ClinVar already
carries the molecular consequence, the gene and an allele frequency, which is the feature set
serving sends, so this avoids running VEP over hundreds of thousands of variants. 2-star
records only.
- train.py now holds out whole genes (GroupShuffleSplit). docs/data.md had said to do this since
the data pass; the code was still doing a random split, which is the leak Grimm 2015 describes.
- evaluate() reports missense on its own. On the last run: AUROC 0.986 over 74,239 held-out
variants, but 0.872 over the 13,553 missense ones, and the docs say plainly why even that is
flattered — within missense the only live feature is allele frequency, and ClinVar's benign
calls often use allele frequency as evidence (ACMG BA1/BS1), so the feature partly caused the
label.
- the 503 now names what is missing (model@alias via tracking URI) and leaves the exception in
the server log instead of the UI.
- make training-set / make train; the 58 MB table is gitignored.
Verified end to end: model registered as v2, the simulated NF2 case scores 0.999 on the planted
variant, and it now ranks 1.00 with all four components live.
Tests: api 77, ml 22, loader 16, web 32; ruff, mypy, svelte-check clean.
An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.
CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)
ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms
Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch
Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.
Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes
Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice
Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.
Kemal Yaylali
kemal