Table of Contents
Architecture
One monorepo holds the scientific pipeline, the API, the interface, model serving and the infrastructure for two deployment tracks.
scientist ──▶ SvelteKit UI ──▶ FastAPI ──▶ PostgreSQL
(/api proxy) │ ▲
│ │
├──▶ MLflow registry (model)
│
└──▶ Nextflow pipeline ──▶ bcftools ──▶ VEP ──▶ loader
The interface proxies /api from its own origin, so there is no CORS anywhere and one public
address serves both.
Why the pieces are where they are
The pipeline is separate from the API on purpose. Annotation takes minutes to hours and needs
containers, reference data and a lot of memory; the API needs to answer in milliseconds. They
communicate through a jobs row and nothing else, which is what lets the same pipeline run as a
local subprocess, a Cloud Run job or an Argo Workflow without the API knowing.
See Execution backends.
Feature engineering travels with the model. The registered MLflow artifact carries
rarelens_ml.features as model code, so serving cannot drift from training. The API sends raw
columns and gets a probability back; it does not know what the features are.
The ontology work happens at load time, not query time. HPO annotations are propagated up the
ontology and each term's information content is computed once by scripts/load-hpo.py, so the
ranking is a set lookup and a weighted sum. See Ranking.
Data model
cases ──┬── case_phenotypes the patient's HPO terms
└── jobs ──── variants ──┬── predictions model score
└── variant_decisions shortlist / dismiss + reason
gene_phenotypes HPO gene→term, propagated (reference data)
hpo_terms term → information content (reference data)
jobs also records what the annotation run produced — has_frequencies,
has_effect_scores. That is not bookkeeping: the ranking uses it to decide which components are
allowed to score at all. See Ranking.
Deployment tracks
| Track | What runs | When to use it |
|---|---|---|
| Serverless (default) | Cloud Run for api + web, Cloud Run job for the Nextflow driver, Google Batch for pipeline tasks | almost always; idles near £1/month |
| Kubernetes (flagged off) | GKE Autopilot, Argo Workflows, Argo Events, ArgoCD | to demonstrate the GitOps path, then destroy |
| Railway | Postgres + api + web, pre-seeded, no pipeline | putting it in front of people quickly |
Both cloud tracks run identical pipeline code; the executor is a Nextflow profile. See Deployments and Infrastructure.
rarelens
Understanding it
Working on it
Running it
When it goes wrong