Kemal Yaylali
1e46fff2ff
docs/blog/rarelens.tex builds a 10-page write-up covering the Nextflow DSL2 pipeline, the three execution backends behind one API call, the Argo and ArgoCD track, Terraform, the external systems integrated, the ranking, the benchmark and the model. It is deliberately as much an account of what was wrong as of what works: the components that scored evidence nobody had looked up, the missense AUROC that fell from 0.872 to 0.500 once allele frequency was removed, the ontology walk that silently dropped 399 terms, and the propagation change that measured slightly worse than what it replaced and was kept anyway with the numbers published. Figures come from one headless-browser script. Screenshots are of the running application; the six diagrams are hand-written HTML styled from the same palette, rendered by the same script. Keeping both in one place is what stops the article drifting from the system, and every number in a diagram has to be changed deliberately.
55 lines
3.7 KiB
HTML
55 lines
3.7 KiB
HTML
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
<div id="frame">
|
|
<h1>rarelens — system architecture</h1>
|
|
<p class="sub">One monorepo: scientific pipeline, API, UI, model serving and two deployment tracks. Arrows are data flow.</p>
|
|
|
|
<div class="row">
|
|
<div class="col" style="flex:0 0 176px">
|
|
<div class="box ink"><div class="t">Clinical scientist</div><div class="d">Enters a proband: a VCF plus the patient's HPO phenotype terms.</div></div>
|
|
<div class="box"><div class="t">SvelteKit UI</div><div class="d">Svelte 5 runes, adapter-node. Same-origin <span class="k">/api</span> proxy, so no CORS.</div><div class="k">web/</div></div>
|
|
</div>
|
|
<div class="arrow">→</div>
|
|
<div class="col" style="flex:0 0 224px">
|
|
<div class="box ink"><div class="t">FastAPI</div><div class="d">Cases, jobs, candidate ranking, decisions, case report. Pydantic v2 + SQLAlchemy 2.0 async.</div><div class="k">api/app/ — 1,380 lines</div></div>
|
|
<div class="box plum"><div class="t">Triage service</div><div class="d">Weighted mean of four auditable components; a component with no evidence abstains.</div><div class="k">app/services/triage.py</div></div>
|
|
</div>
|
|
<div class="arrow">→</div>
|
|
<div class="col" style="flex:0 0 196px">
|
|
<div class="box"><div class="t">PostgreSQL 16</div><div class="d">Cases, phenotypes, jobs, variants, predictions, decisions.</div><div class="k">Alembic migrations</div></div>
|
|
<div class="box muted"><div class="t">HPO reference</div><div class="d">876,585 gene–phenotype annotations after ontology propagation; 11,829 terms with information content.</div></div>
|
|
</div>
|
|
<div class="arrow">→</div>
|
|
<div class="col" style="flex:1">
|
|
<div class="box green"><div class="t">MLflow registry</div><div class="d">LightGBM pyfunc carrying its own feature code, resolved by alias <span class="k">@production</span>. <span class="k">MODEL_URI</span> loads an artifact with no server running.</div><div class="k">ml/</div></div>
|
|
<div class="box amber"><div class="t">Nextflow DSL2 pipeline</div><div class="d">bcftools normalise → Ensembl VEP → loader. Executors: local, Google Batch, Argo Workflows.</div><div class="k">pipeline/</div></div>
|
|
</div>
|
|
</div>
|
|
|
|
<div style="height:16px"></div>
|
|
|
|
<div class="row">
|
|
<div class="band" style="flex:1">
|
|
<div class="lbl">Deployment track A — serverless (default, ~£1/month idle)</div>
|
|
<div class="row">
|
|
<div class="box" style="flex:1"><div class="t">Cloud Run</div><div class="d">api + web, min-instances 0</div></div>
|
|
<div class="arrow">→</div>
|
|
<div class="box" style="flex:1"><div class="t">Cloud Run job</div><div class="d">Nextflow driver, started per run</div></div>
|
|
<div class="arrow">→</div>
|
|
<div class="box" style="flex:1"><div class="t">Google Batch</div><div class="d">VEP tasks on Spot VMs</div></div>
|
|
</div>
|
|
</div>
|
|
<div style="width:14px"></div>
|
|
<div class="band" style="flex:1">
|
|
<div class="lbl">Deployment track B — Kubernetes (behind a Terraform flag)</div>
|
|
<div class="row">
|
|
<div class="box" style="flex:1"><div class="t">Pub/Sub</div><div class="d">vcf-uploaded topic</div></div>
|
|
<div class="arrow">→</div>
|
|
<div class="box" style="flex:1"><div class="t">Argo Events</div><div class="d">sensor → workflow trigger</div></div>
|
|
<div class="arrow">→</div>
|
|
<div class="box" style="flex:1"><div class="t">Argo Workflows</div><div class="d">on GKE Autopilot</div></div>
|
|
</div>
|
|
</div>
|
|
</div>
|
|
<p class="note">Both tracks run the identical pipeline code; the executor is a Nextflow profile, and ArgoCD reconciles the Kubernetes track from the same repository.</p>
|
|
</div>
|