# Architecture

```mermaid
flowchart LR
  U[Scientist] -->|phenotype + VCF| W[SvelteKit web]
  W -->|REST /api| A[FastAPI]
  H[(HPO gene-phenotype annotations)] --> A
  A --> P[(PostgreSQL / Cloud SQL)]
  A -->|publish vcf-uploaded| Q[Pub/Sub]
  Q --> E[Argo Events sensor]
  E --> AW[Argo Workflow: Nextflow driver]
  AW -->|tasks| B[Google Batch: bcftools norm, VEP, load_db]
  B -->|reads VCF, VEP cache| G[(GCS bucket)]
  B -->|writes variants, marks job succeeded| P
  AW -.->|exit handler marks job failed| P
  A -->|rank: phenotype, rarity, consequence, model| C[Ranked candidates -> decisions -> report]
  A -->|models:/rarelens-pathogenicity@production| M[MLflow registry]
  T[ml/train.py] --> M
  GH[GitHub Actions] -->|images via WIF| AR[Artifact Registry]
  GH -->|bumps overlay tags| R[(git: infra/k8s/overlays/gcp)]
  R --> CD[ArgoCD] --> K[GKE Autopilot]
```

## The triage model

A **case** is a proband: a VCF plus the HPO terms observed in that patient. Annotation produces
variants; the model scores them; ranking then answers the only question that matters — which few
variants could explain *this* phenotype.

Rarity (<0.1% in gnomAD) and consequence (HIGH or MODERATE) *filter*, which is the usual first
pass. Phenotype only *ranks*: a real diagnosis can sit in a gene nobody has annotated yet, and
filtering on phenotype would hide exactly that case. The rank is a weighted mean whose parts are
shown next to every candidate (`app/services/triage.py`):

| Component | Weight | Scores when |
|---|---|---|
| share of this patient's phenotype annotated to the gene, weighted by term specificity | 0.35 | always |
| rarity in gnomAD | 0.25 | the run looked up frequencies |
| consequence severity | 0.20 | always |
| model P(pathogenic) | 0.20 | the run has CADD or AlphaMissense |

**A component with no evidence abstains** rather than scoring zero or, worse, full marks. A run
without a VEP cache returns no allele frequencies, and scoring every variant 1.0 for rarity because
nobody consulted gnomAD is a guess wearing the costume of a measurement. The job records what the
run produced, absent components are dropped, and the remaining weights are renormalised so the
score still means the same thing. The model abstains without CADD or AlphaMissense for a related
reason: with only the consequence class it achieves AUROC 0.500 on missense variants, so it would
be restating the consequence component rather than adding evidence. See docs/data.md.

**ClinVar is deliberately not an input.** It sits beside the result as independent confirmation, so
the demo never ranks a variant highly merely because ClinVar already called it pathogenic. On the
simulated NF2 case the planted variant ranks first on phenotype, rarity and consequence alone, and
ClinVar agrees afterwards.

Each candidate can be shortlisted or dismissed with a reason and a note; the case report is that
decision trail plus the funnel counts and the provenance (VEP version, model version, run time).
There is no authentication, so decisions are shared by everyone who opens the demo.

## Two deployment tracks

The same images and the same pipeline, deployed two ways (`infra/terraform/variables.tf`):

| | Serverless (default) | Kubernetes (`-var deploy_kubernetes=true`) |
|---|---|---|
| api, web | Cloud Run, scale to zero | Deployments behind an ingress |
| dispatch | the API executes a Cloud Run job | Pub/Sub -> Argo Events -> Argo Workflow |
| pipeline tasks | Google Batch | Google Batch |
| `/api` routing | the web service proxies it | the ingress routes it |
| idle cost | ~£1/month | ~£130+/month |

`app.services.events.launch()` picks the dispatch backend from configuration: a Cloud Run job when
`CLOUDRUN_JOB` is set, Pub/Sub when `PUBSUB_TOPIC` is, and a local Nextflow process otherwise.
See [cloud.md](cloud.md) for why the serverless one is the default.

## Why these choices

**One monorepo.** The four components share a schema (`variants` table, feature columns) and the
point of the exercise is to see them evolve together. Separate repos would hide the coupling.

**Nextflow for the science, Argo Workflows for the trigger.** Nextflow is the lingua franca for
bioinformatics pipelines. Argo is what the platform team already runs. So Argo owns *when* a
pipeline runs; Nextflow owns *what* it does. The API never talks to Kubernetes directly; it
publishes an event and gets on with its life. The Nextflow driver runs in the Argo pod and sends
each task to Google Batch: a `gs://` work directory needs an executor that stages through GCS
(Nextflow's Kubernetes executor needs a shared ReadWriteMany volume instead).

**Job lifecycle.** The API creates a job as `running` once the pipeline is dispatched, or `failed`
with the reason in `jobs.log` when dispatch is impossible. The loader marks it `succeeded` in the
same transaction that stores the variants. Anything else (Nextflow error, eviction) is caught by
the Argo exit handler, or locally by the API watching the Nextflow process, and marked `failed`,
so the UI never polls a dead job.

**Variant identity.** NORMALISE sets each VCF ID to `CHROM_POS_REF_ALT`; VEP echoes it as
`Uploaded_variation` and the loader takes exact VCF alleles from it, because VEP's own
`Location`/`Allele` columns trim indel alleles.

**FastAPI + Pydantic v2 + SQLAlchemy 2.0 async.** Typed at both boundaries: request/response models
and ORM models are separate on purpose so the database can change without breaking the frontend
contract. Alembic owns the schema; the loader script writes raw SQL against that schema, not the ORM,
because the pipeline container should not import the API.

**SvelteKit.** Small runtime, no virtual DOM, and Svelte 5 runes make server-driven state simple.
The UI has exactly two pages; the goal is a table a scientist actually wants to filter, not a dashboard.
`PUBLIC_API_URL` is read at runtime, so one image works behind the ingress (`/api`) and elsewhere.

**GKE Autopilot + Cloud SQL, not self-managed.** The lab is about the platform patterns (Workload
Identity, GitOps, Kustomize overlays, private networking), not about running etcd. Cloud SQL has only
a private IP; the API reaches it through a Cloud SQL Proxy sidecar, pipeline tasks directly in the VPC.
Database URLs live in Secret Manager.

**GitOps.** CI builds and tests; it never runs `kubectl apply`. It edits image tags in the `gcp` overlay
and ArgoCD reconciles. Rollback is `git revert`. Migrations run in an init container under a Postgres
advisory lock, so replicas starting together migrate once.

**MLflow registry as the model contract.** The API loads whichever version the `production` alias
points at and records that version on every prediction. The registered model is a pyfunc that owns its
feature engineering (`rarelens_ml.features` ships inside it) and returns P(pathogenic), so serving only
sends raw columns and cannot drift from training.

## What is deliberately missing

Authentication, PHI handling, audit logs, clinical validation. This is a learning platform on public
data. Adding Identity-Aware Proxy in front of the ingress is the first step if that ever changes.