kemal
5463f489a3
End-to-end variant interpretation platform for rare genetic disease research: SvelteKit UI, FastAPI + PostgreSQL API, Nextflow/Ensembl VEP pipeline, LightGBM pathogenicity scoring with MLflow, K8s/ArgoCD/GCP infrastructure. Public test data only; no clinical claims.
2.5 KiB
Architecture
flowchart LR
U[Scientist] -->|browser| W[SvelteKit web]
W -->|REST| A[FastAPI]
A --> P[(PostgreSQL / Cloud SQL)]
A -->|publish vcf-uploaded| Q[Pub/Sub]
Q --> E[Argo Events sensor]
E --> AW[Argo Workflow]
AW --> NF[Nextflow: bcftools norm, VEP, load_db]
NF -->|reads VCF| G[(GCS bucket)]
NF -->|writes variants| P
A -->|models:/rarelens-pathogenicity| M[MLflow registry]
T[ml/train.py on GKE, optional GPU] --> M
GH[GitHub Actions] -->|images via WIF| AR[Artifact Registry]
GH -->|bumps overlay tags| R[(git: infra/k8s/overlays/gcp)]
R --> CD[ArgoCD] --> K[GKE Autopilot]
Why these choices
One monorepo. The four components share a schema (variants table, feature columns) and the
point of the exercise is to see them evolve together. Separate repos would hide the coupling.
Nextflow for the science, Argo Workflows for the trigger. Nextflow is the lingua franca for bioinformatics pipelines and has a native Kubernetes executor. Argo is what the platform team already runs. So Argo owns when a pipeline runs; Nextflow owns what it does. The API never talks to Kubernetes directly; it publishes an event and gets on with its life.
FastAPI + Pydantic v2 + SQLAlchemy 2.0 async. Typed at both boundaries: request/response models and ORM models are separate on purpose so the database can change without breaking the frontend contract. Alembic owns the schema; the loader script writes raw SQL against that schema, not the ORM, because the pipeline container should not import the API.
SvelteKit. Small runtime, no virtual DOM, and Svelte 5 runes make server-driven state simple. The UI has exactly two pages; the goal is a table a scientist actually wants to filter, not a dashboard.
GKE Autopilot + Cloud SQL, not self-managed. The lab is about the platform patterns (Workload Identity, GitOps, Kustomize overlays, GPU node selection), not about running etcd.
GitOps. CI builds and tests; it never runs kubectl apply. It edits image tags in the gcp overlay
and ArgoCD reconciles. Rollback is git revert.
MLflow registry as the model contract. The API loads models:/rarelens-pathogenicity/Production.
Training writes there; serving reads there. Feature engineering lives in one module that both sides import.
What is deliberately missing
Authentication, PHI handling, audit logs, clinical validation. This is a learning platform on public data. Adding Identity-Aware Proxy in front of the ingress is the first step if that ever changes.