rarelens/docs/architecture.md

Architecture

flowchart LR
  U[Scientist] -->|phenotype + VCF| W[SvelteKit web]
  W -->|REST /api| A[FastAPI]
  H[(HPO gene-phenotype annotations)] --> A
  A --> P[(PostgreSQL / Cloud SQL)]
  A -->|publish vcf-uploaded| Q[Pub/Sub]
  Q --> E[Argo Events sensor]
  E --> AW[Argo Workflow: Nextflow driver]
  AW -->|tasks| B[Google Batch: bcftools norm, VEP, load_db]
  B -->|reads VCF, VEP cache| G[(GCS bucket)]
  B -->|writes variants, marks job succeeded| P
  AW -.->|exit handler marks job failed| P
  A -->|rank: phenotype, rarity, consequence, model| C[Ranked candidates -> decisions -> report]
  A -->|models:/rarelens-pathogenicity@production| M[MLflow registry]
  T[ml/train.py] --> M
  GH[GitHub Actions] -->|images via WIF| AR[Artifact Registry]
  GH -->|bumps overlay tags| R[(git: infra/k8s/overlays/gcp)]
  R --> CD[ArgoCD] --> K[GKE Autopilot]

The triage model

A case is a proband: a VCF plus the HPO terms observed in that patient. Annotation produces variants; the model scores them; ranking then answers the only question that matters — which few variants could explain this phenotype.

Rarity (<0.1% in gnomAD) and consequence (HIGH or MODERATE) filter, which is the usual first pass. Phenotype only ranks: a real diagnosis can sit in a gene nobody has annotated yet, and filtering on phenotype would hide exactly that case. The rank is a weighted sum whose parts are shown next to every candidate (app/services/triage.py):

Component	Weight
phenotype terms of this patient annotated to the gene	0.35
rarity in gnomAD	0.25
consequence severity	0.20
model P(pathogenic)	0.20

ClinVar is deliberately not an input. It sits beside the result as independent confirmation, so the demo never ranks a variant highly merely because ClinVar already called it pathogenic. On the simulated NF2 case the planted variant ranks first on phenotype, rarity and consequence alone, and ClinVar agrees afterwards.

Each candidate can be shortlisted or dismissed with a reason and a note; the case report is that decision trail plus the funnel counts and the provenance (VEP version, model version, run time). There is no authentication, so decisions are shared by everyone who opens the demo.

Two deployment tracks

The same images and the same pipeline, deployed two ways (infra/terraform/variables.tf):

	Serverless (default)	Kubernetes (-var deploy_kubernetes=true)
api, web	Cloud Run, scale to zero	Deployments behind an ingress
dispatch	the API executes a Cloud Run job	Pub/Sub -> Argo Events -> Argo Workflow
pipeline tasks	Google Batch	Google Batch
/api routing	the web service proxies it	the ingress routes it
idle cost	~£1/month	~£130+/month

app.services.events.launch() picks the dispatch backend from configuration: a Cloud Run job when CLOUDRUN_JOB is set, Pub/Sub when PUBSUB_TOPIC is, and a local Nextflow process otherwise. See cloud.md for why the serverless one is the default.

Why these choices

One monorepo. The four components share a schema (variants table, feature columns) and the point of the exercise is to see them evolve together. Separate repos would hide the coupling.

Nextflow for the science, Argo Workflows for the trigger. Nextflow is the lingua franca for bioinformatics pipelines. Argo is what the platform team already runs. So Argo owns when a pipeline runs; Nextflow owns what it does. The API never talks to Kubernetes directly; it publishes an event and gets on with its life. The Nextflow driver runs in the Argo pod and sends each task to Google Batch: a gs:// work directory needs an executor that stages through GCS (Nextflow's Kubernetes executor needs a shared ReadWriteMany volume instead).

Job lifecycle. The API creates a job as running once the pipeline is dispatched, or failed with the reason in jobs.log when dispatch is impossible. The loader marks it succeeded in the same transaction that stores the variants. Anything else (Nextflow error, eviction) is caught by the Argo exit handler, or locally by the API watching the Nextflow process, and marked failed, so the UI never polls a dead job.

Variant identity. NORMALISE sets each VCF ID to CHROM_POS_REF_ALT; VEP echoes it as Uploaded_variation and the loader takes exact VCF alleles from it, because VEP's own Location/Allele columns trim indel alleles.

FastAPI + Pydantic v2 + SQLAlchemy 2.0 async. Typed at both boundaries: request/response models and ORM models are separate on purpose so the database can change without breaking the frontend contract. Alembic owns the schema; the loader script writes raw SQL against that schema, not the ORM, because the pipeline container should not import the API.

SvelteKit. Small runtime, no virtual DOM, and Svelte 5 runes make server-driven state simple. The UI has exactly two pages; the goal is a table a scientist actually wants to filter, not a dashboard. PUBLIC_API_URL is read at runtime, so one image works behind the ingress (/api) and elsewhere.

GKE Autopilot + Cloud SQL, not self-managed. The lab is about the platform patterns (Workload Identity, GitOps, Kustomize overlays, private networking), not about running etcd. Cloud SQL has only a private IP; the API reaches it through a Cloud SQL Proxy sidecar, pipeline tasks directly in the VPC. Database URLs live in Secret Manager.

GitOps. CI builds and tests; it never runs kubectl apply. It edits image tags in the gcp overlay and ArgoCD reconciles. Rollback is git revert. Migrations run in an init container under a Postgres advisory lock, so replicas starting together migrate once.

MLflow registry as the model contract. The API loads whichever version the production alias points at and records that version on every prediction. The registered model is a pyfunc that owns its feature engineering (rarelens_ml.features ships inside it) and returns P(pathogenic), so serving only sends raw columns and cannot drift from training.

What is deliberately missing

Authentication, PHI handling, audit logs, clinical validation. This is a learning platform on public data. Adding Identity-Aware Proxy in front of the ingress is the first step if that ever changes.