kemal
5463f489a3
ci / api (push) Failing after 10s
ci / terraform (push) Failing after 11s
ci / web (push) Failing after 35s
ci / pipeline (push) Failing after 2m29s
ci / images (api) (push) Skipped
ci / images (ml) (push) Skipped
ci / images (pipeline) (push) Skipped
ci / images (web) (push) Skipped
End-to-end variant interpretation platform for rare genetic disease research: SvelteKit UI, FastAPI + PostgreSQL API, Nextflow/Ensembl VEP pipeline, LightGBM pathogenicity scoring with MLflow, K8s/ArgoCD/GCP infrastructure. Public test data only; no clinical claims.
52 lines
2.5 KiB
Markdown
52 lines
2.5 KiB
Markdown
# Architecture
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
U[Scientist] -->|browser| W[SvelteKit web]
|
|
W -->|REST| A[FastAPI]
|
|
A --> P[(PostgreSQL / Cloud SQL)]
|
|
A -->|publish vcf-uploaded| Q[Pub/Sub]
|
|
Q --> E[Argo Events sensor]
|
|
E --> AW[Argo Workflow]
|
|
AW --> NF[Nextflow: bcftools norm, VEP, load_db]
|
|
NF -->|reads VCF| G[(GCS bucket)]
|
|
NF -->|writes variants| P
|
|
A -->|models:/rarelens-pathogenicity| M[MLflow registry]
|
|
T[ml/train.py on GKE, optional GPU] --> M
|
|
GH[GitHub Actions] -->|images via WIF| AR[Artifact Registry]
|
|
GH -->|bumps overlay tags| R[(git: infra/k8s/overlays/gcp)]
|
|
R --> CD[ArgoCD] --> K[GKE Autopilot]
|
|
```
|
|
|
|
## Why these choices
|
|
|
|
**One monorepo.** The four components share a schema (`variants` table, feature columns) and the
|
|
point of the exercise is to see them evolve together. Separate repos would hide the coupling.
|
|
|
|
**Nextflow for the science, Argo Workflows for the trigger.** Nextflow is the lingua franca for
|
|
bioinformatics pipelines and has a native Kubernetes executor. Argo is what the platform team
|
|
already runs. So Argo owns *when* a pipeline runs; Nextflow owns *what* it does. The API never
|
|
talks to Kubernetes directly; it publishes an event and gets on with its life.
|
|
|
|
**FastAPI + Pydantic v2 + SQLAlchemy 2.0 async.** Typed at both boundaries: request/response models
|
|
and ORM models are separate on purpose so the database can change without breaking the frontend
|
|
contract. Alembic owns the schema; the loader script writes raw SQL against that schema, not the ORM,
|
|
because the pipeline container should not import the API.
|
|
|
|
**SvelteKit.** Small runtime, no virtual DOM, and Svelte 5 runes make server-driven state simple.
|
|
The UI has exactly two pages; the goal is a table a scientist actually wants to filter, not a dashboard.
|
|
|
|
**GKE Autopilot + Cloud SQL, not self-managed.** The lab is about the platform patterns (Workload
|
|
Identity, GitOps, Kustomize overlays, GPU node selection), not about running etcd.
|
|
|
|
**GitOps.** CI builds and tests; it never runs `kubectl apply`. It edits image tags in the `gcp` overlay
|
|
and ArgoCD reconciles. Rollback is `git revert`.
|
|
|
|
**MLflow registry as the model contract.** The API loads `models:/rarelens-pathogenicity/Production`.
|
|
Training writes there; serving reads there. Feature engineering lives in one module that both sides import.
|
|
|
|
## What is deliberately missing
|
|
|
|
Authentication, PHI handling, audit logs, clinical validation. This is a learning platform on public
|
|
data. Adding Identity-Aware Proxy in front of the ingress is the first step if that ever changes.
|