Kemal Yaylali
e76ae847a1
A review of the ranking's arithmetic found four things wrong, all of which made the score look better informed than it was. Measurements below are from this repo, not estimates. **Components now abstain instead of inventing a number.** A run without a VEP cache returns no allele frequencies, and rarity_score(None) read that as "absent from gnomAD, therefore maximally rare" and awarded every variant a free 0.25. jobs.has_frequencies / has_effect_scores record what the run actually produced, absent components are dropped from the weighted mean, and the remaining weights are renormalised so the score keeps its meaning. The UI shows "not looked up" rather than a bar, and the funnel stops calling a step "rare" when nothing was filtered. **Allele frequency is no longer a model feature.** It dominated: the same missense variant scored 0.887 at AF 0 and 0.0003 at AF 0.01. That double- counted, because the ranking already scores frequency explicitly, putting ~45% of every rank on one measurement; and it was circular, because ACMG assigns ClinVar's benign labels using frequency (BA1/BS1). Retraining without it moves missense AUROC from 0.872 to 0.500 — exactly random. The old figure was allele frequency, not variant-effect knowledge. The model therefore abstains unless CADD or AlphaMissense is present, since otherwise it only restates the consequence class. **Phenotype matching is weighted by information content** and HPO annotations are propagated up the ontology. Counting terms alike let "global developmental delay" (IC 0.93) count as much as "dilated left subclavian artery" (IC 7.88). **A real bug in the propagation, found by checking it.** The ancestor walk read a pre-order DFS backwards, which on a DAG lets a term resolve before one of its parents and inherit that parent alone instead of its lineage. It dropped 399 terms out of the phenotype branch, Camptodactyly and Chiari malformation among them. Now a true post-order, tested against a reference transitive closure. The ontology arithmetic moved to rarelens_ml.hpo so it is covered by tests, and rarelens_ml.benchmark measures the whole thing: across 10,178 published cases the causal gene ranks first 45.9-81.0% of the time against 5,269 genes, versus 0.02% for chance. docs/data.md reports that with its contamination (HPO's annotations come from these same case reports), and includes the measurement showing information-content weighting earns its place while propagation does not - kept anyway, for a reason the docs argue rather than assume.
124 lines
7.0 KiB
Markdown
124 lines
7.0 KiB
Markdown
# Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
U[Scientist] -->|phenotype + VCF| W[SvelteKit web]
|
|
W -->|REST /api| A[FastAPI]
|
|
H[(HPO gene-phenotype annotations)] --> A
|
|
A --> P[(PostgreSQL / Cloud SQL)]
|
|
A -->|publish vcf-uploaded| Q[Pub/Sub]
|
|
Q --> E[Argo Events sensor]
|
|
E --> AW[Argo Workflow: Nextflow driver]
|
|
AW -->|tasks| B[Google Batch: bcftools norm, VEP, load_db]
|
|
B -->|reads VCF, VEP cache| G[(GCS bucket)]
|
|
B -->|writes variants, marks job succeeded| P
|
|
AW -.->|exit handler marks job failed| P
|
|
A -->|rank: phenotype, rarity, consequence, model| C[Ranked candidates -> decisions -> report]
|
|
A -->|models:/rarelens-pathogenicity@production| M[MLflow registry]
|
|
T[ml/train.py] --> M
|
|
GH[GitHub Actions] -->|images via WIF| AR[Artifact Registry]
|
|
GH -->|bumps overlay tags| R[(git: infra/k8s/overlays/gcp)]
|
|
R --> CD[ArgoCD] --> K[GKE Autopilot]
|
|
```
|
|
|
|
## The triage model
|
|
|
|
A **case** is a proband: a VCF plus the HPO terms observed in that patient. Annotation produces
|
|
variants; the model scores them; ranking then answers the only question that matters — which few
|
|
variants could explain *this* phenotype.
|
|
|
|
Rarity (<0.1% in gnomAD) and consequence (HIGH or MODERATE) *filter*, which is the usual first
|
|
pass. Phenotype only *ranks*: a real diagnosis can sit in a gene nobody has annotated yet, and
|
|
filtering on phenotype would hide exactly that case. The rank is a weighted mean whose parts are
|
|
shown next to every candidate (`app/services/triage.py`):
|
|
|
|
| Component | Weight | Scores when |
|
|
|---|---|---|
|
|
| share of this patient's phenotype annotated to the gene, weighted by term specificity | 0.35 | always |
|
|
| rarity in gnomAD | 0.25 | the run looked up frequencies |
|
|
| consequence severity | 0.20 | always |
|
|
| model P(pathogenic) | 0.20 | the run has CADD or AlphaMissense |
|
|
|
|
**A component with no evidence abstains** rather than scoring zero or, worse, full marks. A run
|
|
without a VEP cache returns no allele frequencies, and scoring every variant 1.0 for rarity because
|
|
nobody consulted gnomAD is a guess wearing the costume of a measurement. The job records what the
|
|
run produced, absent components are dropped, and the remaining weights are renormalised so the
|
|
score still means the same thing. The model abstains without CADD or AlphaMissense for a related
|
|
reason: with only the consequence class it achieves AUROC 0.500 on missense variants, so it would
|
|
be restating the consequence component rather than adding evidence. See docs/data.md.
|
|
|
|
**ClinVar is deliberately not an input.** It sits beside the result as independent confirmation, so
|
|
the demo never ranks a variant highly merely because ClinVar already called it pathogenic. On the
|
|
simulated NF2 case the planted variant ranks first on phenotype, rarity and consequence alone, and
|
|
ClinVar agrees afterwards.
|
|
|
|
Each candidate can be shortlisted or dismissed with a reason and a note; the case report is that
|
|
decision trail plus the funnel counts and the provenance (VEP version, model version, run time).
|
|
There is no authentication, so decisions are shared by everyone who opens the demo.
|
|
|
|
## Two deployment tracks
|
|
|
|
The same images and the same pipeline, deployed two ways (`infra/terraform/variables.tf`):
|
|
|
|
| | Serverless (default) | Kubernetes (`-var deploy_kubernetes=true`) |
|
|
|---|---|---|
|
|
| api, web | Cloud Run, scale to zero | Deployments behind an ingress |
|
|
| dispatch | the API executes a Cloud Run job | Pub/Sub -> Argo Events -> Argo Workflow |
|
|
| pipeline tasks | Google Batch | Google Batch |
|
|
| `/api` routing | the web service proxies it | the ingress routes it |
|
|
| idle cost | ~£1/month | ~£130+/month |
|
|
|
|
`app.services.events.launch()` picks the dispatch backend from configuration: a Cloud Run job when
|
|
`CLOUDRUN_JOB` is set, Pub/Sub when `PUBSUB_TOPIC` is, and a local Nextflow process otherwise.
|
|
See [cloud.md](cloud.md) for why the serverless one is the default.
|
|
|
|
## Why these choices
|
|
|
|
**One monorepo.** The four components share a schema (`variants` table, feature columns) and the
|
|
point of the exercise is to see them evolve together. Separate repos would hide the coupling.
|
|
|
|
**Nextflow for the science, Argo Workflows for the trigger.** Nextflow is the lingua franca for
|
|
bioinformatics pipelines. Argo is what the platform team already runs. So Argo owns *when* a
|
|
pipeline runs; Nextflow owns *what* it does. The API never talks to Kubernetes directly; it
|
|
publishes an event and gets on with its life. The Nextflow driver runs in the Argo pod and sends
|
|
each task to Google Batch: a `gs://` work directory needs an executor that stages through GCS
|
|
(Nextflow's Kubernetes executor needs a shared ReadWriteMany volume instead).
|
|
|
|
**Job lifecycle.** The API creates a job as `running` once the pipeline is dispatched, or `failed`
|
|
with the reason in `jobs.log` when dispatch is impossible. The loader marks it `succeeded` in the
|
|
same transaction that stores the variants. Anything else (Nextflow error, eviction) is caught by
|
|
the Argo exit handler, or locally by the API watching the Nextflow process, and marked `failed`,
|
|
so the UI never polls a dead job.
|
|
|
|
**Variant identity.** NORMALISE sets each VCF ID to `CHROM_POS_REF_ALT`; VEP echoes it as
|
|
`Uploaded_variation` and the loader takes exact VCF alleles from it, because VEP's own
|
|
`Location`/`Allele` columns trim indel alleles.
|
|
|
|
**FastAPI + Pydantic v2 + SQLAlchemy 2.0 async.** Typed at both boundaries: request/response models
|
|
and ORM models are separate on purpose so the database can change without breaking the frontend
|
|
contract. Alembic owns the schema; the loader script writes raw SQL against that schema, not the ORM,
|
|
because the pipeline container should not import the API.
|
|
|
|
**SvelteKit.** Small runtime, no virtual DOM, and Svelte 5 runes make server-driven state simple.
|
|
The UI has exactly two pages; the goal is a table a scientist actually wants to filter, not a dashboard.
|
|
`PUBLIC_API_URL` is read at runtime, so one image works behind the ingress (`/api`) and elsewhere.
|
|
|
|
**GKE Autopilot + Cloud SQL, not self-managed.** The lab is about the platform patterns (Workload
|
|
Identity, GitOps, Kustomize overlays, private networking), not about running etcd. Cloud SQL has only
|
|
a private IP; the API reaches it through a Cloud SQL Proxy sidecar, pipeline tasks directly in the VPC.
|
|
Database URLs live in Secret Manager.
|
|
|
|
**GitOps.** CI builds and tests; it never runs `kubectl apply`. It edits image tags in the `gcp` overlay
|
|
and ArgoCD reconciles. Rollback is `git revert`. Migrations run in an init container under a Postgres
|
|
advisory lock, so replicas starting together migrate once.
|
|
|
|
**MLflow registry as the model contract.** The API loads whichever version the `production` alias
|
|
points at and records that version on every prediction. The registered model is a pyfunc that owns its
|
|
feature engineering (`rarelens_ml.features` ships inside it) and returns P(pathogenic), so serving only
|
|
sends raw columns and cannot drift from training.
|
|
|
|
## What is deliberately missing
|
|
|
|
Authentication, PHI handling, audit logs, clinical validation. This is a learning platform on public
|
|
data. Adding Identity-Aware Proxy in front of the ingress is the first step if that ever changes.
|