docs: a technical introduction to the platform, as a LaTeX article

docs/blog/rarelens.tex builds a 10-page write-up covering the Nextflow DSL2
pipeline, the three execution backends behind one API call, the Argo and
ArgoCD track, Terraform, the external systems integrated, the ranking, the
benchmark and the model.

It is deliberately as much an account of what was wrong as of what works: the
components that scored evidence nobody had looked up, the missense AUROC that
fell from 0.872 to 0.500 once allele frequency was removed, the ontology walk
that silently dropped 399 terms, and the propagation change that measured
slightly worse than what it replaced and was kept anyway with the numbers
published.

Figures come from one headless-browser script. Screenshots are of the running
application; the six diagrams are hand-written HTML styled from the same
palette, rendered by the same script. Keeping both in one place is what stops
the article drifting from the system, and every number in a diagram has to be
changed deliberately.
This commit is contained in:
Kemal Yaylali
2026-09-12 13:10:47 +01:00
parent c25fb53666
commit 1e46fff2ff
25 changed files with 729 additions and 0 deletions
+7
View File
@@ -19,4 +19,11 @@ data/*.vcf*
data/*.tsv data/*.tsv
data/*.case.json data/*.case.json
data/*.zip data/*.zip
# LaTeX build artefacts; the PDF itself is committed as the deliverable.
docs/blog/*.aux
docs/blog/*.log
docs/blog/*.out
docs/blog/preview-*.png
docs/blog/diagram-sources/node_modules/
docs/blog/diagram-sources/out/
!data/README.md !data/README.md
+26
View File
@@ -0,0 +1,26 @@
# Build the technical introduction. Needs a LaTeX toolchain; the Docker route needs none installed.
#
# make # build rarelens.pdf
# make figures # re-capture screenshots and re-render the diagrams (needs the app running)
# make clean
TEX ?= docker run --rm --platform linux/amd64 -v "$(PWD):/w" -w /w ghcr.io/xu-cheng/texlive-full pdflatex
BASE ?= http://localhost:5173
CHROME ?= /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
rarelens.pdf: rarelens.tex $(wildcard figures/*.png)
$(TEX) -interaction=nonstopmode rarelens.tex >/dev/null
$(TEX) -interaction=nonstopmode rarelens.tex | grep -E 'Output written|^!' || true
# Screenshots come from the running app, diagrams from the HTML in diagram-sources/, both through
# one headless browser so the figures and the interface cannot drift apart.
figures:
cd diagram-sources && npm install --silent playwright && \
BASE="$(BASE)" CHROME_PATH="$(CHROME)" \
DIAGRAMS=arch,events,pipeline,evidence,benchmark,model node shoot.mjs
cp diagram-sources/out/*.png figures/
clean:
rm -f rarelens.aux rarelens.log rarelens.out rarelens.toc preview-*.png
.PHONY: figures clean
+28
View File
@@ -0,0 +1,28 @@
# The technical introduction
`rarelens.tex` is a standalone technical write-up of the platform — pipeline, event-driven
execution, infrastructure, the ranking, the model, and the parts that turned out to be wrong.
`rarelens.pdf` is the built article (10 pages).
```bash
make # build the PDF (uses a Docker TeX Live; nothing to install locally)
make figures # re-capture every figure, then rebuild
make clean
```
## Where the figures come from
Both kinds of figure are produced by one headless-browser script, `diagram-sources/shoot.mjs`, so
the article cannot drift away from the thing it describes:
- **Screenshots** (`01``06`) are taken from the running application. `make figures` needs it up —
`make up && make migrate && make hpo`, the API on `:8000`, the UI on `:5173` — and a case that
has been analysed. The published LoeysDietz case is the default; override with `CASE_ID`.
- **Diagrams** (`arch`, `events`, `pipeline`, `evidence`, `benchmark`, `model`) are HTML pages in
`diagram-sources/`, styled from the same palette as the application, rendered to PNG by the same
script. They are hand-written rather than generated, so any number in them has to be updated
deliberately when the measurement changes — which is the point.
Figures are committed because rebuilding them needs a running stack and a browser. If a
measurement in the article changes, change it in the diagram source, re-run `make figures`, and
say so in the commit.
+33
View File
@@ -0,0 +1,33 @@
:root {
--paper:#f3f5f7; --ink:#16243a; --soft:#52627a; --line:#cfd6df;
--plum:#7a1f5c; --plum-soft:#f0dbe8; --amber:#b86a00; --green:#1f6b4a; --green-soft:#dcece4;
--mono: ui-monospace, 'SF Mono', Menlo, monospace;
--sans: system-ui, -apple-system, 'Segoe UI', sans-serif;
}
* { box-sizing: border-box; }
body { margin:0; background:#fff; font-family:var(--sans); color:var(--ink); }
#frame { background:#fff; padding:28px 30px; width:1180px; }
h1 { font-size:19px; font-weight:600; margin:0 0 4px; letter-spacing:-0.01em; }
.sub { color:var(--soft); font-size:13px; margin:0 0 22px; }
.row { display:flex; gap:14px; align-items:stretch; }
.col { display:flex; flex-direction:column; gap:14px; }
.box { border:1.5px solid var(--line); border-radius:7px; background:#fff; padding:11px 13px; }
.box .t { font-weight:600; font-size:13.5px; letter-spacing:-0.005em; }
.box .d { color:var(--soft); font-size:11.5px; margin-top:3px; line-height:1.45; }
.box .k { font-family:var(--mono); font-size:10.5px; color:var(--soft); margin-top:5px; }
.ink { border-color:var(--ink); }
.plum { border-color:var(--plum); background:#fdf7fb; }
.plum .t { color:var(--plum); }
.green { border-color:var(--green); background:#f4faf7; }
.green .t { color:var(--green); }
.amber { border-color:var(--amber); background:#fffaf2; }
.amber .t { color:var(--amber); }
.muted { background:var(--paper); }
.band { border:1.5px dashed var(--line); border-radius:9px; padding:13px; background:#fcfdfe; }
.band > .lbl { font-size:10.5px; text-transform:uppercase; letter-spacing:0.09em; color:var(--soft); font-weight:600; margin-bottom:9px; }
.arrow { display:flex; align-items:center; justify-content:center; color:var(--soft); font-size:17px; padding:0 2px; }
.arrow.down { justify-content:flex-start; padding:1px 0 1px 18px; }
.note { font-size:11px; color:var(--soft); font-style:italic; margin-top:9px; }
.pill { display:inline-block; font-size:10.5px; font-family:var(--mono); padding:1.5px 6px; border-radius:10px; background:var(--paper); color:var(--soft); border:1px solid var(--line); }
.pill.on { background:var(--green-soft); color:var(--green); border-color:var(--green); }
.pill.off { background:var(--plum-soft); color:var(--plum); border-color:var(--plum); }
+54
View File
@@ -0,0 +1,54 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<div id="frame">
<h1>rarelens — system architecture</h1>
<p class="sub">One monorepo: scientific pipeline, API, UI, model serving and two deployment tracks. Arrows are data flow.</p>
<div class="row">
<div class="col" style="flex:0 0 176px">
<div class="box ink"><div class="t">Clinical scientist</div><div class="d">Enters a proband: a VCF plus the patient's HPO phenotype terms.</div></div>
<div class="box"><div class="t">SvelteKit UI</div><div class="d">Svelte 5 runes, adapter-node. Same-origin <span class="k">/api</span> proxy, so no CORS.</div><div class="k">web/</div></div>
</div>
<div class="arrow">&#8594;</div>
<div class="col" style="flex:0 0 224px">
<div class="box ink"><div class="t">FastAPI</div><div class="d">Cases, jobs, candidate ranking, decisions, case report. Pydantic v2 + SQLAlchemy 2.0 async.</div><div class="k">api/app/ — 1,380 lines</div></div>
<div class="box plum"><div class="t">Triage service</div><div class="d">Weighted mean of four auditable components; a component with no evidence abstains.</div><div class="k">app/services/triage.py</div></div>
</div>
<div class="arrow">&#8594;</div>
<div class="col" style="flex:0 0 196px">
<div class="box"><div class="t">PostgreSQL 16</div><div class="d">Cases, phenotypes, jobs, variants, predictions, decisions.</div><div class="k">Alembic migrations</div></div>
<div class="box muted"><div class="t">HPO reference</div><div class="d">876,585 gene&#8211;phenotype annotations after ontology propagation; 11,829 terms with information content.</div></div>
</div>
<div class="arrow">&#8594;</div>
<div class="col" style="flex:1">
<div class="box green"><div class="t">MLflow registry</div><div class="d">LightGBM pyfunc carrying its own feature code, resolved by alias <span class="k">@production</span>. <span class="k">MODEL_URI</span> loads an artifact with no server running.</div><div class="k">ml/</div></div>
<div class="box amber"><div class="t">Nextflow DSL2 pipeline</div><div class="d">bcftools normalise &#8594; Ensembl VEP &#8594; loader. Executors: local, Google Batch, Argo Workflows.</div><div class="k">pipeline/</div></div>
</div>
</div>
<div style="height:16px"></div>
<div class="row">
<div class="band" style="flex:1">
<div class="lbl">Deployment track A &mdash; serverless (default, ~&pound;1/month idle)</div>
<div class="row">
<div class="box" style="flex:1"><div class="t">Cloud Run</div><div class="d">api + web, min-instances 0</div></div>
<div class="arrow">&#8594;</div>
<div class="box" style="flex:1"><div class="t">Cloud Run job</div><div class="d">Nextflow driver, started per run</div></div>
<div class="arrow">&#8594;</div>
<div class="box" style="flex:1"><div class="t">Google Batch</div><div class="d">VEP tasks on Spot VMs</div></div>
</div>
</div>
<div style="width:14px"></div>
<div class="band" style="flex:1">
<div class="lbl">Deployment track B &mdash; Kubernetes (behind a Terraform flag)</div>
<div class="row">
<div class="box" style="flex:1"><div class="t">Pub/Sub</div><div class="d">vcf-uploaded topic</div></div>
<div class="arrow">&#8594;</div>
<div class="box" style="flex:1"><div class="t">Argo Events</div><div class="d">sensor &#8594; workflow trigger</div></div>
<div class="arrow">&#8594;</div>
<div class="box" style="flex:1"><div class="t">Argo Workflows</div><div class="d">on GKE Autopilot</div></div>
</div>
</div>
</div>
<p class="note">Both tracks run the identical pipeline code; the executor is a Nextflow profile, and ArgoCD reconciles the Kubernetes track from the same repository.</p>
</div>
+37
View File
@@ -0,0 +1,37 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<style>
.bar { height:19px; border-radius:3px; background:var(--plum); }
.bar.pes { background:var(--plum-soft); border:1px solid var(--plum); }
.bar.base { background:var(--soft); }
.brow { display:grid; grid-template-columns:210px 1fr 62px; align-items:center; gap:11px; margin-bottom:7px; font-size:12.5px; }
.brow .lab { color:var(--soft); text-align:right; }
.brow .val { font-family:var(--mono); font-size:11.5px; }
.track { background:var(--paper); border-radius:3px; }
</style>
<div id="frame">
<h1>Does the phenotype ranking actually work?</h1>
<p class="sub">Every case in Monarch's Phenopacket Store: given a real patient's reported terms, where does the gene their authors diagnosed rank among all 5,269 HPO-annotated genes? Ties give a range &mdash; optimistic counts a tie as a win, pessimistic counts every tied gene as ahead.</p>
<div style="font-size:12px;color:var(--soft);font-weight:600;text-transform:uppercase;letter-spacing:0.08em;margin:4px 0 9px">Causal gene ranked first (top-1), 10,178 published cases</div>
<div class="brow"><div class="lab">optimistic</div><div class="track"><div class="bar" style="width:81.0%"></div></div><div class="val">81.0%</div></div>
<div class="brow"><div class="lab">pessimistic</div><div class="track"><div class="bar pes" style="width:45.9%"></div></div><div class="val">45.9%</div></div>
<div class="brow"><div class="lab">random baseline</div><div class="track"><div class="bar base" style="width:0.4%"></div></div><div class="val">0.02%</div></div>
<div style="font-size:12px;color:var(--soft);font-weight:600;text-transform:uppercase;letter-spacing:0.08em;margin:20px 0 9px">Causal gene in the top ten</div>
<div class="brow"><div class="lab">optimistic</div><div class="track"><div class="bar" style="width:87.2%"></div></div><div class="val">87.2%</div></div>
<div class="brow"><div class="lab">pessimistic</div><div class="track"><div class="bar pes" style="width:69.7%"></div></div><div class="val">69.7%</div></div>
<div style="height:22px"></div>
<div class="row">
<div class="box amber" style="flex:1"><div class="t">The benchmark is contaminated, and it must be said out loud</div><div class="d">The median causal gene already carries <em>every one</em> of its patient's terms, because HPO's gene annotations are curated from these same case reports. This measures how well the ranking retrieves a gene HPO has already been told about: an upper bound. A prospective number, on a patient whose gene nobody has annotated yet, would be lower, and this corpus cannot say by how much.</div></div>
<div style="width:15px"></div>
<div class="box" style="flex:1"><div class="t">Measuring my own changes, including the one that failed</div><div class="d">Information-content weighting and ontology propagation both replaced plain term counting. Asked whether they helped, the corpus said only one of them did &mdash; pessimistic figures, 6,485 cases with six or more terms:</div>
<table style="width:100%;border-collapse:collapse;font-size:11.5px;margin-top:9px">
<tr style="color:var(--soft)"><td style="padding:3px 0">count terms (original)</td><td style="text-align:right;font-family:var(--mono)">61.8%</td><td style="text-align:right;font-family:var(--mono)">0.682</td></tr>
<tr style="color:var(--green);font-weight:600"><td style="padding:3px 0">+ information content</td><td style="text-align:right;font-family:var(--mono)">63.6%</td><td style="text-align:right;font-family:var(--mono)">0.706</td></tr>
<tr style="color:var(--soft)"><td style="padding:3px 0">+ propagation</td><td style="text-align:right;font-family:var(--mono)">58.2%</td><td style="text-align:right;font-family:var(--mono)">0.653</td></tr>
<tr><td style="padding:3px 0">+ both (shipped)</td><td style="text-align:right;font-family:var(--mono)">59.5%</td><td style="text-align:right;font-family:var(--mono)">0.670</td></tr>
</table>
<div class="d" style="margin-top:7px">Weighting earns its place. Propagation costs about what weighting gains &mdash; kept for a reason the documentation argues rather than assumes, with the table there so a reader can disagree.</div></div>
</div>
</div>
+22
View File
@@ -0,0 +1,22 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<div id="frame">
<h1>One API call, three execution backends</h1>
<p class="sub"><span class="k" style="font-family:var(--mono)">POST /cases/{id}/annotate</span> creates a job row, then <span class="k" style="font-family:var(--mono)">events.launch()</span> picks a backend from configuration alone. The pipeline code never changes.</p>
<div class="row" style="align-items:flex-start">
<div class="box ink" style="flex:0 0 210px"><div class="t">events.launch(job)</div><div class="d">Chooses on settings, in order. Everything downstream writes back to the same <span class="k" style="font-family:var(--mono)">jobs</span> row, so the UI polls one endpoint whichever path ran.</div><div class="k">app/services/events.py</div></div>
<div class="arrow">&#8594;</div>
<div class="col" style="flex:1">
<div class="box amber"><div class="t">1. cloudrun_job set &#8594; Cloud Run job</div><div class="d">Executes the Nextflow driver as a job with argument overrides, via the Jobs API. Scales to zero between runs; the service account holds only <span class="k" style="font-family:var(--mono)">run.jobsExecutorWithOverrides</span> on that one job.</div></div>
<div class="box green"><div class="t">2. pubsub_topic set &#8594; Pub/Sub publish</div><div class="d">Publishes the job as an event. Argo Events' sensor subscribes and triggers an Argo Workflow on GKE. Decouples the API from the compute entirely; retries and back-pressure belong to the queue.</div></div>
<div class="box plum"><div class="t">3. otherwise &#8594; local subprocess</div><div class="d">Runs Nextflow directly and streams stdout into the job log, so the UI can show live progress. This is what a developer gets with no cloud configured at all.</div></div>
</div>
<div class="arrow">&#8594;</div>
<div class="col" style="flex:0 0 215px">
<div class="box"><div class="t">Nextflow pipeline</div><div class="d">NORMALISE &#8594; VEP &#8594; LOAD_DB</div></div>
<div class="arrow down">&#8595;</div>
<div class="box"><div class="t">loader writes results</div><div class="d">Variants inserted, job marked succeeded, and the run records <em>what evidence it looked up</em>.</div><div class="k">has_frequencies, has_effect_scores</div></div>
</div>
</div>
<p class="note">The database URL never appears on a command line: it is passed by environment, or as a Nextflow secret, so it stays out of <span class="k" style="font-family:var(--mono)">.command.sh</span> and the workflow logs.</p>
</div>
+30
View File
@@ -0,0 +1,30 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<div id="frame">
<h1>Evidence that was never looked up must abstain</h1>
<p class="sub">The rank is a weighted mean of four auditable components. Which ones may score is decided per job, from what the annotation run actually produced.</p>
<div class="row">
<div class="box plum" style="flex:1"><div class="t">phenotype &mdash; 0.35</div><div class="d">Share of the patient's HPO terms annotated to the gene, each weighted by its information content.</div><div class="k">always scores</div></div>
<div class="box" style="flex:1"><div class="t">rarity &mdash; 0.25</div><div class="d">Allele frequency in gnomAD, as a step function a reviewer can read.</div><div class="k">only if the run looked up frequencies</div></div>
<div class="box" style="flex:1"><div class="t">consequence &mdash; 0.20</div><div class="d">Predicted impact severity: HIGH, MODERATE, LOW, MODIFIER.</div><div class="k">always scores</div></div>
<div class="box" style="flex:1"><div class="t">model &mdash; 0.20</div><div class="d">LightGBM P(pathogenic) from variant effect.</div><div class="k">only with CADD or AlphaMissense</div></div>
</div>
<div style="height:20px"></div>
<div class="row">
<div class="band" style="flex:1">
<div class="lbl">Before &mdash; the bug</div>
<div class="box" style="border-color:var(--plum)"><div class="t" style="color:var(--plum)">rarity_score(None) &#8594; 1.00</div><div class="d">"No frequency in this run" was read as "absent from gnomAD, therefore maximally rare", and every variant collected a free 0.25. The model, meanwhile, scored 0.887 on features it had never seen. Two of four components were fiction, and the score looked fully informed.</div></div>
</div>
<div style="width:16px"></div>
<div class="band" style="flex:1">
<div class="lbl">After &mdash; abstention and renormalisation</div>
<div class="box" style="border-color:var(--green)"><div class="t" style="color:var(--green)">component &#8594; null, weights renormalised</div><div class="d">The job records <span class="k" style="font-family:var(--mono)">has_frequencies</span> and <span class="k" style="font-family:var(--mono)">has_effect_scores</span>. Absent components drop out and the remaining weights renormalise over 0.55, so the score stays on a 0&ndash;1 scale and still means the same thing. The UI prints "not looked up", never a bar.</div></div>
</div>
</div>
<div style="height:18px"></div>
<table style="width:100%; border-collapse:collapse; font-size:12.5px; border:1px solid var(--line)">
<tr style="background:var(--paper)"><th style="text-align:left;padding:7px 10px;border-bottom:1px solid var(--line)">Published Loeys&ndash;Dietz case, run without a VEP cache</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">score</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">phenotype (0.64)</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">rarity</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">consequence (0.36)</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">model</th></tr>
<tr><td style="padding:7px 10px;border-bottom:1px solid var(--line)"><em>TGFBR2</em> 3:30672252 G&gt;T missense &mdash; the published diagnosis</td><td style="padding:7px 10px;text-align:center;border-bottom:1px solid var(--line)"><strong style="color:var(--plum)">0.855</strong></td><td style="padding:7px 10px;text-align:center;border-bottom:1px solid var(--line)">1.00 <span style="color:var(--soft)">(30/30)</span></td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic;border-bottom:1px solid var(--line)">not looked up</td><td style="padding:7px 10px;text-align:center;border-bottom:1px solid var(--line)">0.60</td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic;border-bottom:1px solid var(--line)">not looked up</td></tr>
<tr><td style="padding:7px 10px"><em>OSBPL10</em> 3:31748090 missense &mdash; incidental</td><td style="padding:7px 10px;text-align:center">0.218</td><td style="padding:7px 10px;text-align:center">0.00</td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic">not looked up</td><td style="padding:7px 10px;text-align:center">0.60</td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic">not looked up</td></tr>
</table>
<p class="note">Both are rare missense variants, identical on every piece of evidence this run holds except one. The phenotype is what separates a published diagnosis from an incidental variant in a lipid-transport gene &mdash; which is the argument for phenotype-driven triage, in one table.</p>
</div>
+25
View File
@@ -0,0 +1,25 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<style>
.big { font-family:var(--mono); font-size:34px; font-weight:600; letter-spacing:-0.02em; }
.mrow { display:grid; grid-template-columns:1fr 92px 92px 110px 110px; font-size:13px; align-items:center; }
.mrow > div { padding:8px 10px; border-bottom:1px solid var(--line); }
.mhead > div { color:var(--soft); font-weight:600; font-size:11px; text-transform:uppercase; letter-spacing:0.06em; background:var(--paper); }
.num { font-family:var(--mono); text-align:right; }
</style>
<div id="frame">
<h1>The number that changed what the model is allowed to do</h1>
<p class="sub">Held-out evaluation with whole genes held out, never single variants (Grimm <em>et al.</em> 2015): 312,025 training and 74,239 test variants across 7,728 and 1,932 genes, with no gene on both sides.</p>
<div class="mrow mhead"><div>Model</div><div class="num">AUROC</div><div class="num">AUPRC</div><div class="num">missense AUROC</div><div class="num">missense AUPRC</div></div>
<div class="mrow"><div>v2 &mdash; with gnomAD allele frequency as a feature</div><div class="num">0.986</div><div class="num">0.954</div><div class="num">0.872</div><div class="num">0.725</div></div>
<div class="mrow"><div>v3 &mdash; allele frequency removed</div><div class="num">0.966</div><div class="num">0.881</div><div class="num" style="color:var(--plum);font-weight:700">0.500</div><div class="num">0.398</div></div>
<div style="height:20px"></div>
<div class="row">
<div class="box plum" style="flex:0 0 268px; text-align:center; padding:17px"><div class="big" style="color:var(--plum)">0.500</div><div class="d" style="margin-top:5px">AUROC on missense variants once frequency is removed. Exactly random.</div></div>
<div style="width:15px"></div>
<div class="col" style="flex:1">
<div class="box"><div class="t">What that proves</div><div class="d">Strip frequency out and the model cannot tell one missense variant from another at all &mdash; nothing is left but the consequence class, so every missense row scores identically. The respectable-looking 0.872 was never variant-effect knowledge. It was allele frequency.</div></div>
<div class="box amber"><div class="t">And the frequency feature was circular</div><div class="d">ACMG's BA1/BS1 criteria assign ClinVar's <em>benign</em> labels <em>using</em> allele frequency. The feature had partly caused the label, so the model was rediscovering the rule that produced its own training data.</div></div>
<div class="box green"><div class="t">The consequence for the product</div><div class="d">Frequency is no longer a feature &mdash; the ranking already scores it explicitly and auditably, and feeding it to the model as well put ~45% of every rank on one measurement counted twice. The model now abstains unless it has CADD or AlphaMissense, because 0.500 is the measurement saying it has nothing else to add.</div></div>
</div>
</div>
</div>
+18
View File
@@ -0,0 +1,18 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<div id="frame">
<h1>The Nextflow DSL2 pipeline</h1>
<p class="sub">Three processes, each a container. The same workflow file runs on a laptop, on Google Batch and under Argo Workflows &mdash; the executor is a profile, not a rewrite.</p>
<div class="row" style="align-items:stretch">
<div class="box" style="flex:1"><div class="t">NORMALISE</div><div class="d">bcftools norm: left-align and split multi-allelics, then set the VCF ID to <span class="k" style="font-family:var(--mono)">CHROM_POS_REF_ALT</span>.</div><div class="k">bcftools 1.20</div></div>
<div class="arrow">&#8594;</div>
<div class="box amber" style="flex:1.25"><div class="t">VEP</div><div class="d">Ensembl VEP 113. Cache mode for full annotation, or <span class="k" style="font-family:var(--mono)">--database</span> against Ensembl's public server when the 25&nbsp;GB cache is not available. CADD and AlphaMissense are optional plugins.</div><div class="k">4 cpus, 8 GB</div></div>
<div class="arrow">&#8594;</div>
<div class="box" style="flex:1"><div class="t">LOAD_DB</div><div class="d">Parses VEP <span class="k" style="font-family:var(--mono)">--tab</span> output, inserts variants idempotently (a retried task cannot duplicate), and marks the job succeeded.</div><div class="k">psycopg, chunked COPY</div></div>
</div>
<div style="height:18px"></div>
<div class="row">
<div class="box plum" style="flex:1"><div class="t">Why the ID carries identity</div><div class="d">VEP's own Location and Allele columns trim indel alleles and shift positions, so a deletion round-trips as the wrong variant. Carrying <span class="k" style="font-family:var(--mono)">CHROM_POS_REF_ALT</span> through the ID field keeps the exact alleles the caller emitted. Verified on a real run: <span class="k" style="font-family:var(--mono)">22:42126611 CT&gt;C</span> survives intact.</div></div>
<div style="width:14px"></div>
<div class="box green" style="flex:1"><div class="t">Stub blocks keep CI honest</div><div class="d">Every process has a <span class="k" style="font-family:var(--mono)">stub:</span> block, so CI runs <span class="k" style="font-family:var(--mono)">nextflow run main.nf -stub-run</span> on a three-record fixture and checks wiring and channel shapes on every pull request &mdash; no containers, no VEP cache, seconds not hours.</div></div>
</div>
</div>
+67
View File
@@ -0,0 +1,67 @@
import { chromium } from 'playwright';
import { mkdirSync } from 'node:fs';
import { resolve } from 'node:path';
const BASE = process.env.BASE ?? 'http://localhost:5173';
const CASE = process.env.CASE_ID ?? '91d85d3d-3351-4a7d-8d29-164cdea76518';
const OUT = resolve('out');
mkdirSync(OUT, { recursive: true });
const shot = async (page, name, opts = {}) => {
await page.screenshot({ path: `${OUT}/${name}.png`, ...opts });
console.log('wrote', name);
};
const EXE = process.env.CHROME_PATH;
const browser = await chromium.launch(EXE ? { executablePath: EXE } : {});
const page = await browser.newPage({
viewport: { width: 1440, height: 1000 },
deviceScaleFactor: 2
});
// 1. case list
await page.goto(`${BASE}/`, { waitUntil: 'networkidle' });
await page.waitForSelector('.caselist li', { timeout: 20000 });
await shot(page, '01-cases', { fullPage: true });
// 2. the published case: funnel, filters, ranked candidates
await page.goto(`${BASE}/cases/${CASE}`, { waitUntil: 'networkidle' });
await page.waitForSelector('.candidate', { timeout: 30000 });
await page.waitForTimeout(600);
await shot(page, '02-case', { fullPage: true });
// 3. the evidence panel for the top candidate: the components table, abstentions and all
await page.locator('.candidate').first().click();
await page.waitForSelector('.panel', { timeout: 20000 });
await page.waitForTimeout(600);
await shot(page, '03-variant-panel', { fullPage: true });
// just the panel, cropped, for a tighter figure
const panel = page.locator('.panel').first();
await panel.screenshot({ path: `${OUT}/04-panel-only.png` });
console.log('wrote 04-panel-only');
// just the funnel
const funnel = page.locator('.funnel, [class*="funnel"]').first();
if (await funnel.count()) {
await funnel.screenshot({ path: `${OUT}/05-funnel.png` });
console.log('wrote 05-funnel');
}
// 4. the case report
await page.goto(`${BASE}/cases/${CASE}/report`, { waitUntil: 'networkidle' });
await page.waitForTimeout(1200);
await shot(page, '06-report', { fullPage: true });
// 5. local diagram pages rendered to PNG
for (const name of process.env.DIAGRAMS?.split(',').filter(Boolean) ?? []) {
await page.goto(`file://${resolve(`${name}.html`)}`, {
waitUntil: 'networkidle'
});
await page.waitForTimeout(300);
const box = page.locator('#frame');
await box.screenshot({ path: `${OUT}/${name}.png` });
console.log('wrote', name);
}
await browser.close();
Binary file not shown.

After

Width:  |  Height:  |  Size: 180 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 301 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 611 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 276 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 26 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 340 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 279 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 254 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 233 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 287 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 222 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 188 KiB

Binary file not shown.
+382
View File
@@ -0,0 +1,382 @@
% rarelens - a technical introduction
% Build: make -C docs/blog (or: pdflatex rarelens.tex, twice)
\documentclass[11pt,a4paper]{article}
\usepackage[T1]{fontenc}
\usepackage[utf8]{inputenc}
\usepackage{lmodern}
\usepackage{microtype}
\usepackage[margin=2.4cm,top=2.6cm,bottom=2.6cm]{geometry}
\usepackage{graphicx}
\usepackage{booktabs}
\usepackage{xcolor}
\usepackage{caption}
\usepackage{enumitem}
\usepackage{fancyhdr}
\usepackage{titlesec}
\usepackage{listings}
\usepackage{float}
\usepackage[hidelinks]{hyperref}
\definecolor{ink}{HTML}{16243A}
\definecolor{soft}{HTML}{52627A}
\definecolor{plum}{HTML}{7A1F5C}
\definecolor{line}{HTML}{CFD6DF}
\definecolor{paper}{HTML}{F3F5F7}
\hypersetup{colorlinks=true,linkcolor=plum,urlcolor=plum,citecolor=plum}
\color{ink}
\titleformat{\section}{\normalfont\Large\bfseries\color{ink}}{\thesection}{0.7em}{}
\titleformat{\subsection}{\normalfont\large\bfseries\color{ink}}{\thesubsection}{0.6em}{}
\titlespacing*{\section}{0pt}{1.6em}{0.6em}
\captionsetup{font=small,labelfont={bf,color=soft},textfont={color=soft},width=0.94\textwidth}
\setlist[itemize]{leftmargin=1.2em,itemsep=0.25em,topsep=0.35em}
\setlength{\parskip}{0.55em}
\setlength{\parindent}{0pt}
\pagestyle{fancy}
\fancyhf{}
\fancyhead[L]{\small\color{soft}rarelens --- a technical introduction}
\fancyfoot[C]{\small\color{soft}\thepage}
\renewcommand{\headrulewidth}{0.4pt}
\lstdefinestyle{code}{
basicstyle=\ttfamily\footnotesize\color{ink},
backgroundcolor=\color{paper},
frame=single, rulecolor=\color{line}, framesep=5pt,
breaklines=true, showstringspaces=false,
keywordstyle=\color{plum}, commentstyle=\color{soft}\itshape,
xleftmargin=0pt, aboveskip=0.9em, belowskip=0.9em
}
\lstset{style=code}
% LaTeX is conservative about letting text share a page with a large float, which on a document
% this figure-heavy strands them on near-empty pages of their own. Loosen the thresholds.
\renewcommand{\topfraction}{0.92}
\renewcommand{\bottomfraction}{0.80}
\renewcommand{\textfraction}{0.06}
\renewcommand{\floatpagefraction}{0.82}
\setcounter{topnumber}{2}
\setcounter{totalnumber}{3}
% htbp, not H: a full-width figure that will not fit pinned in place leaves half a page blank.
\newcommand{\fig}[3]{%
\begin{figure}[htbp]\centering
\includegraphics[width=#2\textwidth]{#1}
\caption{#3}\end{figure}}
\begin{document}
\begin{center}
{\LARGE\bfseries rarelens}\\[0.35em]
{\large\color{soft} An end-to-end rare-disease variant triage platform}\\[1.1em]
{\color{soft}\small A technical introduction --- pipeline, infrastructure, model and the evidence behind the ranking}
\end{center}
\vspace{0.6em}
\hrule
\vspace{1.2em}
\section*{In one paragraph}
\textbf{rarelens takes a patient's genome and their clinical phenotype and narrows thousands of
variants to a handful a scientist can actually review, showing the evidence behind every rank.}
It is a working monorepo: a Nextflow DSL2 pipeline that annotates variants with Ensembl VEP, a
FastAPI service that ranks them against Human Phenotype Ontology annotations, a LightGBM model
served from an MLflow registry, a SvelteKit interface built around triage decisions rather than
table filtering, and Terraform for two deployment tracks on Google Cloud --- a serverless one that
idles at roughly \pounds1 a month, and a Kubernetes one with Argo Workflows, Argo Events and
ArgoCD behind a feature flag. Everything runs on public, openly licensed data. No patient data is
used, accepted, or possible to load.
This document is a technical tour. It is also, deliberately, an account of the things that turned
out to be wrong --- including a model metric that collapsed from 0.872 to 0.500 the moment I took
away the feature that was doing all the work, and an ontology bug that silently deleted 399 terms.
\section{The problem}
A rare-disease proband's exome contains something like twenty thousand coding variants. Perhaps a
few hundred are rare and protein-altering. Exactly one, usually, explains why the patient is ill.
Finding it is not a filtering problem so much as an evidence problem: the same variant is
uninteresting in one patient and diagnostic in another, and what changes between them is the
clinical phenotype.
That observation drives the whole design. The interface is not a variant table with filters --- it
is a ranked shortlist where each candidate carries the four pieces of evidence that put it there,
and where a reviewer shortlists or dismisses with a reason that ends up in a case report.
\fig{figures/arch.png}{1.0}{The system. One repository holds the pipeline, the API, the interface,
model serving and the infrastructure for two deployment tracks. The scientist never sees any of it
except the middle column.}
\section{The Nextflow pipeline}
Annotation is a three-process DSL2 workflow, each process a pinned container: \texttt{bcftools}
normalises, Ensembl VEP 113 annotates, and a loader writes results into PostgreSQL and marks the
job succeeded.
\fig{figures/pipeline.png}{1.0}{The pipeline. The same workflow file runs on a laptop, on Google
Batch and under Argo Workflows; the executor is a profile, not a rewrite.}
Two details are worth pulling out, because both cost real debugging time.
\subsection{Variant identity survives annotation}
VEP's \texttt{Location} and \texttt{Allele} output columns trim indel alleles and shift positions
by one, which is correct for VEP's own purposes and wrong as a primary key. A deletion written by
the caller as \texttt{CT>C} comes back as \texttt{->} at a different coordinate, and the loader
then stores a variant that does not exist in the input file.
The fix is to carry identity through a field VEP does not touch. \texttt{NORMALISE} sets the VCF ID
to \texttt{CHROM\_POS\_REF\_ALT}, and the loader parses identity back out of it:
\begin{lstlisting}[language=bash]
bcftools norm -m -any -f $genome | \
bcftools annotate --set-id '%CHROM\_%POS\_%REF\_%FIRST_ALT'
\end{lstlisting}
Verified on a real run: \texttt{22:42126611 CT>C} round-trips with its alleles intact.
\subsection{Stub blocks make continuous integration possible at all}
A VEP cache is 25\,GB. No pull request is going to download one. Every process therefore carries a
\texttt{stub:} block, and CI runs the workflow with \texttt{-stub-run} against a three-record
fixture, checking channel wiring and process contracts in seconds without a container or a cache in
sight. It catches the failure that actually happens in practice --- a renamed output, a channel
that emits the wrong cardinality --- while leaving the scientific correctness to the tests that can
afford to be slow.
\section{Event-driven execution, three ways}
\texttt{POST /cases/\{id\}/annotate} writes a job row and hands off. What happens next is a
configuration decision, not a code path the caller knows about.
\fig{figures/events.png}{1.0}{One entry point, three backends, chosen from settings alone.}
\begin{itemize}
\item \textbf{Cloud Run job.} The Nextflow driver runs as a Cloud Run job started through the
Jobs API with argument overrides. It scales to zero between runs, and its service account
holds \texttt{run.jobsExecutorWithOverrides} on exactly one job --- not project-wide.
\item \textbf{Pub/Sub.} The job is published as an event. An Argo Events sensor subscribes and
triggers an Argo Workflow on GKE. This is the decoupled path: retries, ordering and
back-pressure become the queue's problem rather than the API's, and the API can be
restarted mid-pipeline without losing work.
\item \textbf{Local subprocess.} Nextflow runs directly and its stdout is streamed into the job
log, so the interface shows live progress. This is what a developer gets with nothing
configured, and it is the same code path the other two wrap.
\end{itemize}
All three converge on the same \texttt{jobs} row, so the interface polls one endpoint regardless.
The database URL is passed by environment or as a Nextflow secret and never appears on a command
line, keeping it out of \texttt{.command.sh} and the workflow logs.
\section{Kubernetes, Argo and GitOps}
The Kubernetes track is Kustomize bases with a local overlay (kind, an in-cluster Postgres) and a
GCP overlay (Cloud SQL, Workload Identity). Argo Workflows runs the annotation
\texttt{WorkflowTemplate}; Argo Events holds the Pub/Sub EventSource and the sensor that triggers
it; ArgoCD reconciles the cluster from the repository, and a green CI run on \texttt{main} bumps
image tags in the GCP overlay so that deployment is a commit rather than a command.
One bug from this area is worth recording because it is invisible until it bites: Kustomize
generates hashed ConfigMap names so that a configuration change forces a rollout, but the hashed
name is only substituted into workloads Kustomize believes are in scope. The overlays were missing
\texttt{namespace: rarelens}, so the substitution silently did not happen and pods mounted a
ConfigMap name that no longer existed. The symptom was a pod stuck in \texttt{CreateContainerConfigError}
with nothing wrong in the manifests as written.
\section{Infrastructure as code}
Terraform provisions both tracks from one root module, with the expensive half behind flags:
\begin{lstlisting}[language=bash]
terraform apply -var project=<id> # serverless: Cloud Run + Batch
terraform apply -var project=<id> -var deploy_kubernetes=true \
-var deploy_cloud_sql=true # adds GKE, Argo, Cloud SQL
\end{lstlisting}
The default track provisions Cloud Run services for the API and interface, a Cloud Run job for the
Nextflow driver, Google Batch for pipeline tasks on Spot VMs, a GCS bucket, Secret Manager entries
and Artifact Registry. GKE Autopilot and Cloud SQL are opt-in, because a Kubernetes control plane
and a managed database are most of what a demonstration estate costs.
CI authenticates to Google Cloud through Workload Identity Federation, so there is no service
account key anywhere in the repository or in GitHub secrets. Terraform is validated and
format-checked on every pull request.
\subsection{Cost as a design constraint}
A portfolio platform is idle more than 99\% of the time, which makes idle cost the only cost that
matters. The serverless track is built around that: Cloud Run at \texttt{min-instances=0}, a driver
that exists only while a pipeline runs, and Batch on Spot. Idle cost lands near \pounds1 a month,
almost all of it the database. The Kubernetes track exists to demonstrate the GitOps path and is
meant to be destroyed afterwards.
\section{Integrating external systems}
Almost nothing here is self-contained, and integrating public biological infrastructure is most of
the work:
\begin{itemize}
\item \textbf{Ensembl VEP} in cache mode, or against Ensembl's \emph{public database server}
with \texttt{--database} when 25\,GB is not available --- slower per variant, no plugin
scores, but no download.
\item \textbf{gnomAD v4.1} allele frequencies streamed by genomic region straight out of the
public Google Cloud bucket. The files are tabix-indexed, so a range request returns a few
hundred records without fetching the file. I had previously written off frequencies as
impossible without the full cache; testing that assumption disproved it.
\item \textbf{Human Phenotype Ontology} gene-to-phenotype annotations plus the ontology itself,
propagated and weighted at load time.
\item \textbf{ClinVar} for model training labels, filtered to two-star review status and above.
\item \textbf{Phenopacket Store} (Monarch Initiative), which curates published case reports into
GA4GH phenopackets --- the source of both the demonstration case and the benchmark.
\item \textbf{MLflow} as a model registry, resolved by alias, with a registry-free path for
deployments that should not run a tracking server.
\end{itemize}
\begin{lstlisting}[language=bash]
# real gnomAD frequencies with no bulk download, verified end to end
bcftools view -r chr3:30672000-30673000 \
https://storage.googleapis.com/gcp-public-data--gnomad/release/4.1/...chr3.vcf.bgz
# -> 392 records, 5 KB, then fed to VEP with --custom alongside --database
\end{lstlisting}
\section{The ranking, and what it refuses to claim}
The rank is a weighted mean of four components a reviewer can audit. ClinVar is deliberately
\emph{not} one of them --- it sits beside the result as independent confirmation, so nothing ranks
highly merely because ClinVar already called it pathogenic.
\fig{figures/evidence.png}{1.0}{Rarity and consequence filter; phenotype only ranks, because a real
diagnosis can sit in a gene nobody has annotated yet. A component with no evidence behind it
abstains, and the remaining weights renormalise.}
The abstention rule replaced a genuine bug. Run without a VEP cache, there are no allele
frequencies --- and the code read a missing frequency as \emph{absent from gnomAD, therefore
maximally rare}, handing every variant a free quarter of its score. The model, separately, was
returning 0.887 for every variant on features it had never been given. Two of four components were
fiction, and the total looked fully informed.
The fix is structural rather than cosmetic: the loader records what the annotation run actually
produced (\texttt{has\_frequencies}, \texttt{has\_effect\_scores}), components without evidence
return null instead of a number, and the weights renormalise over whatever is left. The interface
prints ``not looked up'' where it would otherwise have drawn a bar.
\section{Measuring the ranking}
One demonstration case ranking correctly is an anecdote. The benchmark asks the only question that
matters for a phenotype-driven tool, across every usable case in Phenopacket Store.
\fig{figures/benchmark.png}{1.0}{Retrieval against 5,269 candidate genes, and an honest report of
what the measurement cannot tell you.}
Two things in that figure matter more than the headline. The first is the contamination: HPO's gene
annotations are curated from these same case reports, so the median causal gene already carries
every one of its patient's terms. The number is an upper bound and is labelled as one. The second
is that when I measured my own improvements, one of them did not work --- information-content
weighting helped, ontology propagation cost about as much as weighting gained. That result is in
the documentation with the table, rather than quietly dropped.
\section{The model, and the number that changed it}
The pathogenicity model is LightGBM trained on ClinVar labels, held out \emph{by gene} rather than
by variant. That distinction is not pedantry: a random split puts variants of the same gene on both
sides, and the model then scores the gene instead of the variant, which is exactly the inflation
Grimm \emph{et al.} documented for this class of tool in 2015.
\fig{figures/model.png}{1.0}{Removing one feature moved missense AUROC from 0.872 to 0.500.}
The model originally took allele frequency as a feature and looked respectable. Probing it showed
frequency dominating everything --- the same missense variant scored 0.887 at frequency zero and
0.0003 at one per cent. That is two separate problems. It double-counted, because the ranking
already scores frequency explicitly, putting roughly 45\% of every rank on one measurement. And it
was circular, because ACMG's BA1/BS1 criteria assign ClinVar's benign labels \emph{using} allele
frequency, so the model was rediscovering the rule that had generated its own labels.
Retraining without it returned missense AUROC 0.500 --- exactly random. With frequency gone and no
CADD or AlphaMissense scores in the training table, nothing is left but the consequence class, so
every missense variant scores identically. The conclusion is unambiguous and slightly
uncomfortable: the model never had variant-effect knowledge. It now abstains from the ranking
unless it has a predictor the other components do not already provide, and 0.500 is the measurement
that justifies the abstention.
\section{The application}
The interface is built around the narrowing and the decision, not around the table.
\fig{figures/02-case.png}{1.0}{A published case. The funnel across the top is the story --- variants
called, rarity, coding candidates, phenotype-matched --- and each candidate carries its evidence as
chips. The note under the header states plainly which components did not score and why.}
\fig{figures/04-panel-only.png}{0.62}{Selecting a candidate opens the arithmetic. Every component is
shown with its weight and its contribution, and the two that had nothing to go on say so rather
than displaying a zero.}
\fig{figures/06-report.png}{0.92}{The case report: shortlisted and dismissed variants with the
reviewer's reasons, the funnel counts, and provenance --- VEP version, model version, run time ---
so a result can be reproduced or challenged later.}
\section{Engineering practice}
\begin{itemize}
\item \textbf{Tests where the risk is.} 99 Python tests plus 49 in the front end, concentrated on
the ranking arithmetic, the ontology handling, the loader's idempotency and the security
boundary. Database tests run against a real PostgreSQL --- an embedded server locally, a
service container in CI --- because the schema is part of the behaviour.
\item \textbf{Migrations are tested, both ways.} Alembic upgrade and downgrade are exercised
against a live database, which is how a downgrade that left a stale enum type behind was
caught.
\item \textbf{CI runs the whole estate:} lint and types for the API, unit tests for the model
code, a real Postgres for the loader, a Nextflow stub run, and \texttt{terraform validate}.
\item \textbf{Input validation as a security boundary.} A case's \texttt{vcf\_uri} must be a
\texttt{gs://} object or an absolute path beneath a configured data root, with a VCF
suffix. That is what stops a crafted path becoming a Nextflow option or reading an
arbitrary file, and it has its own test file.
\end{itemize}
\section{Deployment}
The demonstration runs on Railway --- three services, with the API reachable only on the private
network so the interface's \texttt{/api} proxy is the single public entry point. One shared
credential sits in front of it.
Railway cannot run the pipeline: Nextflow shells out to \texttt{docker run} for VEP and bcftools,
and a container platform gives you a container, not a Docker daemon. Rather than leave a button
that always fails, the cases are annotated where the pipeline works and copied up, and the
pipeline actions are hidden. Making it idle correctly needed one real change --- the platform
decides a service is idle from its \emph{outbound} traffic, and a pooled database connection is
outbound traffic, so the default connection pool would have kept the service awake and billable
for ever.
\section{What is still wrong}
An honest introduction should end with the open problems, not the achievements.
\begin{itemize}
\item \textbf{The model has no features in its training data.} Its only two remaining inputs,
CADD and AlphaMissense, are absent from all 688,362 training rows. It abstains today, so
nothing is broken --- but installing the VEP plugins would flip it on while it still knows
nothing. Serving should refuse unless the model was trained on the features it is handed.
\item \textbf{The benchmark is contaminated} and no amount of code fixes it. A leave-one-publication-out
rebuild is possible --- HPO's annotation file carries the source PMID ---
and is the honest next step.
\item \textbf{No baseline comparison.} Scoring the same held-out rows with CADD and AlphaMissense
would give the model something to beat. AlphaMissense is a 0.64\,GB download and tractable;
CADD's whole-genome file is 87.5\,GB and is not.
\item \textbf{Rarity contributes nothing without the cache} in the default demonstration, though
the gnomAD streaming route above now makes that solvable.
\end{itemize}
\vspace{1.4em}
\hrule
\vspace{0.9em}
{\small\color{soft}
rarelens is a self-training project built in the open on public data, licensed AGPL-3.0. It is not
a clinical tool and makes no diagnostic claim: ACMG/AMP treats computational predictions as
supporting evidence only, never sufficient alone. Data sources, licences, citations and the
evaluation caveats are documented in \texttt{docs/data.md}; the architecture decisions, including
why Google Cloud rather than AWS, are in \texttt{docs/architecture.md} and \texttt{docs/cloud.md}.}
\end{document}