Kemal Yaylali
07a01715fd
A table with filters made the user do the work. Rare disease triage is a different task:
which few variants could explain *this* patient's phenotype, and why. The app now answers
that, and lets a reviewer act on the answer.
Domain
- a case is a proband: a VCF plus the HPO terms observed in the patient (samples -> cases)
- HPO's gene-to-phenotype annotations are loaded as reference data (scripts/load-hpo.py)
- each candidate can be shortlisted or dismissed with a reason and a note
Ranking (app/services/triage.py, 21 tests)
- weighted sum of phenotype match, rarity, consequence severity and the model's score,
with every component shown next to the candidate
- rarity and consequence filter; phenotype only ranks, because a real diagnosis can sit in
a gene nobody has annotated yet and filtering on it would hide exactly that case
- ClinVar is deliberately not an input: it appears beside the result as independent
confirmation, so nothing ranks highly merely because ClinVar already said pathogenic
UI
- the funnel is the headline: variants called -> rare -> coding candidates -> phenotype-matched
- ranked candidates with evidence chips, not a grid of everything; filters are demoted
- a variant panel showing the score breakdown, the matched HPO terms, the raw VEP record and
links out to Ensembl/gnomAD/ClinVar, with the decision controls
- a printable case report: phenotype, funnel, shortlisted variants with reasons, provenance
API: /cases with phenotypes, /cases/{id}/candidates (funnel + ranked + weights),
/variants/{id}, /variants/{id}/decision, /cases/{id}/report, /phenotypes for the picker.
Scoring moved under the case and now answers 503 with the reason when no model registry is
reachable, instead of a 500.
Verified end to end on a simulated proband (scripts/make-demo-case.sh: real GIAB HG002
background + one real ClinVar 2-star pathogenic NF2 variant). 13 variants called -> 1 coding
candidate, and the planted variant ranks first at 0.80 on phenotype 1.00, rarity 1.00 and
consequence 1.00, with ClinVar agreeing afterwards.
Tests: api 75, ml 18, loader 16, web 27; ruff, mypy, svelte-check, terraform validate, both
kustomize overlays and the Nextflow stub run all clean.
116 lines
6.2 KiB
Markdown
116 lines
6.2 KiB
Markdown
# Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
U[Scientist] -->|phenotype + VCF| W[SvelteKit web]
|
|
W -->|REST /api| A[FastAPI]
|
|
H[(HPO gene-phenotype annotations)] --> A
|
|
A --> P[(PostgreSQL / Cloud SQL)]
|
|
A -->|publish vcf-uploaded| Q[Pub/Sub]
|
|
Q --> E[Argo Events sensor]
|
|
E --> AW[Argo Workflow: Nextflow driver]
|
|
AW -->|tasks| B[Google Batch: bcftools norm, VEP, load_db]
|
|
B -->|reads VCF, VEP cache| G[(GCS bucket)]
|
|
B -->|writes variants, marks job succeeded| P
|
|
AW -.->|exit handler marks job failed| P
|
|
A -->|rank: phenotype, rarity, consequence, model| C[Ranked candidates -> decisions -> report]
|
|
A -->|models:/rarelens-pathogenicity@production| M[MLflow registry]
|
|
T[ml/train.py] --> M
|
|
GH[GitHub Actions] -->|images via WIF| AR[Artifact Registry]
|
|
GH -->|bumps overlay tags| R[(git: infra/k8s/overlays/gcp)]
|
|
R --> CD[ArgoCD] --> K[GKE Autopilot]
|
|
```
|
|
|
|
## The triage model
|
|
|
|
A **case** is a proband: a VCF plus the HPO terms observed in that patient. Annotation produces
|
|
variants; the model scores them; ranking then answers the only question that matters — which few
|
|
variants could explain *this* phenotype.
|
|
|
|
Rarity (<0.1% in gnomAD) and consequence (HIGH or MODERATE) *filter*, which is the usual first
|
|
pass. Phenotype only *ranks*: a real diagnosis can sit in a gene nobody has annotated yet, and
|
|
filtering on phenotype would hide exactly that case. The rank is a weighted sum whose parts are
|
|
shown next to every candidate (`app/services/triage.py`):
|
|
|
|
| Component | Weight |
|
|
|---|---|
|
|
| phenotype terms of this patient annotated to the gene | 0.35 |
|
|
| rarity in gnomAD | 0.25 |
|
|
| consequence severity | 0.20 |
|
|
| model P(pathogenic) | 0.20 |
|
|
|
|
**ClinVar is deliberately not an input.** It sits beside the result as independent confirmation, so
|
|
the demo never ranks a variant highly merely because ClinVar already called it pathogenic. On the
|
|
simulated NF2 case the planted variant ranks first on phenotype, rarity and consequence alone, and
|
|
ClinVar agrees afterwards.
|
|
|
|
Each candidate can be shortlisted or dismissed with a reason and a note; the case report is that
|
|
decision trail plus the funnel counts and the provenance (VEP version, model version, run time).
|
|
There is no authentication, so decisions are shared by everyone who opens the demo.
|
|
|
|
## Two deployment tracks
|
|
|
|
The same images and the same pipeline, deployed two ways (`infra/terraform/variables.tf`):
|
|
|
|
| | Serverless (default) | Kubernetes (`-var deploy_kubernetes=true`) |
|
|
|---|---|---|
|
|
| api, web | Cloud Run, scale to zero | Deployments behind an ingress |
|
|
| dispatch | the API executes a Cloud Run job | Pub/Sub -> Argo Events -> Argo Workflow |
|
|
| pipeline tasks | Google Batch | Google Batch |
|
|
| `/api` routing | the web service proxies it | the ingress routes it |
|
|
| idle cost | ~£1/month | ~£130+/month |
|
|
|
|
`app.services.events.launch()` picks the dispatch backend from configuration: a Cloud Run job when
|
|
`CLOUDRUN_JOB` is set, Pub/Sub when `PUBSUB_TOPIC` is, and a local Nextflow process otherwise.
|
|
See [cloud.md](cloud.md) for why the serverless one is the default.
|
|
|
|
## Why these choices
|
|
|
|
**One monorepo.** The four components share a schema (`variants` table, feature columns) and the
|
|
point of the exercise is to see them evolve together. Separate repos would hide the coupling.
|
|
|
|
**Nextflow for the science, Argo Workflows for the trigger.** Nextflow is the lingua franca for
|
|
bioinformatics pipelines. Argo is what the platform team already runs. So Argo owns *when* a
|
|
pipeline runs; Nextflow owns *what* it does. The API never talks to Kubernetes directly; it
|
|
publishes an event and gets on with its life. The Nextflow driver runs in the Argo pod and sends
|
|
each task to Google Batch: a `gs://` work directory needs an executor that stages through GCS
|
|
(Nextflow's Kubernetes executor needs a shared ReadWriteMany volume instead).
|
|
|
|
**Job lifecycle.** The API creates a job as `running` once the pipeline is dispatched, or `failed`
|
|
with the reason in `jobs.log` when dispatch is impossible. The loader marks it `succeeded` in the
|
|
same transaction that stores the variants. Anything else (Nextflow error, eviction) is caught by
|
|
the Argo exit handler, or locally by the API watching the Nextflow process, and marked `failed`,
|
|
so the UI never polls a dead job.
|
|
|
|
**Variant identity.** NORMALISE sets each VCF ID to `CHROM_POS_REF_ALT`; VEP echoes it as
|
|
`Uploaded_variation` and the loader takes exact VCF alleles from it, because VEP's own
|
|
`Location`/`Allele` columns trim indel alleles.
|
|
|
|
**FastAPI + Pydantic v2 + SQLAlchemy 2.0 async.** Typed at both boundaries: request/response models
|
|
and ORM models are separate on purpose so the database can change without breaking the frontend
|
|
contract. Alembic owns the schema; the loader script writes raw SQL against that schema, not the ORM,
|
|
because the pipeline container should not import the API.
|
|
|
|
**SvelteKit.** Small runtime, no virtual DOM, and Svelte 5 runes make server-driven state simple.
|
|
The UI has exactly two pages; the goal is a table a scientist actually wants to filter, not a dashboard.
|
|
`PUBLIC_API_URL` is read at runtime, so one image works behind the ingress (`/api`) and elsewhere.
|
|
|
|
**GKE Autopilot + Cloud SQL, not self-managed.** The lab is about the platform patterns (Workload
|
|
Identity, GitOps, Kustomize overlays, private networking), not about running etcd. Cloud SQL has only
|
|
a private IP; the API reaches it through a Cloud SQL Proxy sidecar, pipeline tasks directly in the VPC.
|
|
Database URLs live in Secret Manager.
|
|
|
|
**GitOps.** CI builds and tests; it never runs `kubectl apply`. It edits image tags in the `gcp` overlay
|
|
and ArgoCD reconciles. Rollback is `git revert`. Migrations run in an init container under a Postgres
|
|
advisory lock, so replicas starting together migrate once.
|
|
|
|
**MLflow registry as the model contract.** The API loads whichever version the `production` alias
|
|
points at and records that version on every prediction. The registered model is a pyfunc that owns its
|
|
feature engineering (`rarelens_ml.features` ships inside it) and returns P(pathogenic), so serving only
|
|
sends raw columns and cannot drift from training.
|
|
|
|
## What is deliberately missing
|
|
|
|
Authentication, PHI handling, audit logs, clinical validation. This is a learning platform on public
|
|
data. Adding Identity-Aware Proxy in front of the ingress is the first step if that ever changes.
|