chore: keep the write-up out of the repository
ci / api (push) Failing after 2s
ci / pipeline (push) Failing after 1s
ci / ml (push) Failing after 8s
ci / terraform (push) Successful in 13s
ci / web (push) Successful in 44s
ci / images (api, api, api/Dockerfile) (push) Skipped
ci / images (loader, pipeline, pipeline/loader.Dockerfile) (push) Skipped
ci / images (ml, ml, ml/Dockerfile) (push) Skipped
ci / images (pipeline, pipeline, pipeline/Dockerfile) (push) Skipped
ci / images (web, web, web/Dockerfile) (push) Skipped
docs/blog/ is untracked and ignored. The article, its figures and the diagram sources stay in the working directory - the PDF is still the thing to send - but 6 MB of rendered output does not belong in the history, and the post now lives as a WordPress draft with its own copies of the images.
@@ -19,12 +19,7 @@ data/*.vcf*
|
|||||||
data/*.tsv
|
data/*.tsv
|
||||||
data/*.case.json
|
data/*.case.json
|
||||||
data/*.zip
|
data/*.zip
|
||||||
# LaTeX build artefacts; the PDF itself is committed as the deliverable.
|
# The write-up and its figures are kept locally but out of the repository.
|
||||||
docs/blog/*.aux
|
docs/blog/
|
||||||
docs/blog/*.log
|
|
||||||
docs/blog/*.out
|
|
||||||
docs/blog/preview-*.png
|
|
||||||
docs/blog/diagram-sources/node_modules/
|
|
||||||
docs/blog/diagram-sources/out/
|
|
||||||
!data/README.md
|
!data/README.md
|
||||||
.DS_Store
|
.DS_Store
|
||||||
|
|||||||
@@ -1,26 +0,0 @@
|
|||||||
# Build the technical introduction. Needs a LaTeX toolchain; the Docker route needs none installed.
|
|
||||||
#
|
|
||||||
# make # build rarelens.pdf
|
|
||||||
# make figures # re-capture screenshots and re-render the diagrams (needs the app running)
|
|
||||||
# make clean
|
|
||||||
|
|
||||||
TEX ?= docker run --rm --platform linux/amd64 -v "$(PWD):/w" -w /w ghcr.io/xu-cheng/texlive-full pdflatex
|
|
||||||
BASE ?= http://localhost:5173
|
|
||||||
CHROME ?= /Applications/Google Chrome.app/Contents/MacOS/Google Chrome
|
|
||||||
|
|
||||||
rarelens.pdf: rarelens.tex $(wildcard figures/*.png)
|
|
||||||
$(TEX) -interaction=nonstopmode rarelens.tex >/dev/null
|
|
||||||
$(TEX) -interaction=nonstopmode rarelens.tex | grep -E 'Output written|^!' || true
|
|
||||||
|
|
||||||
# Screenshots come from the running app, diagrams from the HTML in diagram-sources/, both through
|
|
||||||
# one headless browser so the figures and the interface cannot drift apart.
|
|
||||||
figures:
|
|
||||||
cd diagram-sources && npm install --silent playwright && \
|
|
||||||
BASE="$(BASE)" CHROME_PATH="$(CHROME)" \
|
|
||||||
DIAGRAMS=arch,events,pipeline,evidence,benchmark,model node shoot.mjs
|
|
||||||
cp diagram-sources/out/*.png figures/
|
|
||||||
|
|
||||||
clean:
|
|
||||||
rm -f rarelens.aux rarelens.log rarelens.out rarelens.toc preview-*.png
|
|
||||||
|
|
||||||
.PHONY: figures clean
|
|
||||||
@@ -1,28 +0,0 @@
|
|||||||
# The technical introduction
|
|
||||||
|
|
||||||
`rarelens.tex` is a standalone technical write-up of the platform — pipeline, event-driven
|
|
||||||
execution, infrastructure, the ranking, the model, and the parts that turned out to be wrong.
|
|
||||||
`rarelens.pdf` is the built article (10 pages).
|
|
||||||
|
|
||||||
```bash
|
|
||||||
make # build the PDF (uses a Docker TeX Live; nothing to install locally)
|
|
||||||
make figures # re-capture every figure, then rebuild
|
|
||||||
make clean
|
|
||||||
```
|
|
||||||
|
|
||||||
## Where the figures come from
|
|
||||||
|
|
||||||
Both kinds of figure are produced by one headless-browser script, `diagram-sources/shoot.mjs`, so
|
|
||||||
the article cannot drift away from the thing it describes:
|
|
||||||
|
|
||||||
- **Screenshots** (`01`–`06`) are taken from the running application. `make figures` needs it up —
|
|
||||||
`make up && make migrate && make hpo`, the API on `:8000`, the UI on `:5173` — and a case that
|
|
||||||
has been analysed. The published Loeys–Dietz case is the default; override with `CASE_ID`.
|
|
||||||
- **Diagrams** (`arch`, `events`, `pipeline`, `evidence`, `benchmark`, `model`) are HTML pages in
|
|
||||||
`diagram-sources/`, styled from the same palette as the application, rendered to PNG by the same
|
|
||||||
script. They are hand-written rather than generated, so any number in them has to be updated
|
|
||||||
deliberately when the measurement changes — which is the point.
|
|
||||||
|
|
||||||
Figures are committed because rebuilding them needs a running stack and a browser. If a
|
|
||||||
measurement in the article changes, change it in the diagram source, re-run `make figures`, and
|
|
||||||
say so in the commit.
|
|
||||||
@@ -1,33 +0,0 @@
|
|||||||
:root {
|
|
||||||
--paper:#f3f5f7; --ink:#16243a; --soft:#52627a; --line:#cfd6df;
|
|
||||||
--plum:#7a1f5c; --plum-soft:#f0dbe8; --amber:#b86a00; --green:#1f6b4a; --green-soft:#dcece4;
|
|
||||||
--mono: ui-monospace, 'SF Mono', Menlo, monospace;
|
|
||||||
--sans: system-ui, -apple-system, 'Segoe UI', sans-serif;
|
|
||||||
}
|
|
||||||
* { box-sizing: border-box; }
|
|
||||||
body { margin:0; background:#fff; font-family:var(--sans); color:var(--ink); }
|
|
||||||
#frame { background:#fff; padding:28px 30px; width:1180px; }
|
|
||||||
h1 { font-size:19px; font-weight:600; margin:0 0 4px; letter-spacing:-0.01em; }
|
|
||||||
.sub { color:var(--soft); font-size:13px; margin:0 0 22px; }
|
|
||||||
.row { display:flex; gap:14px; align-items:stretch; }
|
|
||||||
.col { display:flex; flex-direction:column; gap:14px; }
|
|
||||||
.box { border:1.5px solid var(--line); border-radius:7px; background:#fff; padding:11px 13px; }
|
|
||||||
.box .t { font-weight:600; font-size:13.5px; letter-spacing:-0.005em; }
|
|
||||||
.box .d { color:var(--soft); font-size:11.5px; margin-top:3px; line-height:1.45; }
|
|
||||||
.box .k { font-family:var(--mono); font-size:10.5px; color:var(--soft); margin-top:5px; }
|
|
||||||
.ink { border-color:var(--ink); }
|
|
||||||
.plum { border-color:var(--plum); background:#fdf7fb; }
|
|
||||||
.plum .t { color:var(--plum); }
|
|
||||||
.green { border-color:var(--green); background:#f4faf7; }
|
|
||||||
.green .t { color:var(--green); }
|
|
||||||
.amber { border-color:var(--amber); background:#fffaf2; }
|
|
||||||
.amber .t { color:var(--amber); }
|
|
||||||
.muted { background:var(--paper); }
|
|
||||||
.band { border:1.5px dashed var(--line); border-radius:9px; padding:13px; background:#fcfdfe; }
|
|
||||||
.band > .lbl { font-size:10.5px; text-transform:uppercase; letter-spacing:0.09em; color:var(--soft); font-weight:600; margin-bottom:9px; }
|
|
||||||
.arrow { display:flex; align-items:center; justify-content:center; color:var(--soft); font-size:17px; padding:0 2px; }
|
|
||||||
.arrow.down { justify-content:flex-start; padding:1px 0 1px 18px; }
|
|
||||||
.note { font-size:11px; color:var(--soft); font-style:italic; margin-top:9px; }
|
|
||||||
.pill { display:inline-block; font-size:10.5px; font-family:var(--mono); padding:1.5px 6px; border-radius:10px; background:var(--paper); color:var(--soft); border:1px solid var(--line); }
|
|
||||||
.pill.on { background:var(--green-soft); color:var(--green); border-color:var(--green); }
|
|
||||||
.pill.off { background:var(--plum-soft); color:var(--plum); border-color:var(--plum); }
|
|
||||||
@@ -1,54 +0,0 @@
|
|||||||
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
||||||
<div id="frame">
|
|
||||||
<h1>rarelens — system architecture</h1>
|
|
||||||
<p class="sub">One monorepo: scientific pipeline, API, UI, model serving and two deployment tracks. Arrows are data flow.</p>
|
|
||||||
|
|
||||||
<div class="row">
|
|
||||||
<div class="col" style="flex:0 0 176px">
|
|
||||||
<div class="box ink"><div class="t">Clinical scientist</div><div class="d">Enters a proband: a VCF plus the patient's HPO phenotype terms.</div></div>
|
|
||||||
<div class="box"><div class="t">SvelteKit UI</div><div class="d">Svelte 5 runes, adapter-node. Same-origin <span class="k">/api</span> proxy, so no CORS.</div><div class="k">web/</div></div>
|
|
||||||
</div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="col" style="flex:0 0 224px">
|
|
||||||
<div class="box ink"><div class="t">FastAPI</div><div class="d">Cases, jobs, candidate ranking, decisions, case report. Pydantic v2 + SQLAlchemy 2.0 async.</div><div class="k">api/app/ — 1,380 lines</div></div>
|
|
||||||
<div class="box plum"><div class="t">Triage service</div><div class="d">Weighted mean of four auditable components; a component with no evidence abstains.</div><div class="k">app/services/triage.py</div></div>
|
|
||||||
</div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="col" style="flex:0 0 196px">
|
|
||||||
<div class="box"><div class="t">PostgreSQL 16</div><div class="d">Cases, phenotypes, jobs, variants, predictions, decisions.</div><div class="k">Alembic migrations</div></div>
|
|
||||||
<div class="box muted"><div class="t">HPO reference</div><div class="d">876,585 gene–phenotype annotations after ontology propagation; 11,829 terms with information content.</div></div>
|
|
||||||
</div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="col" style="flex:1">
|
|
||||||
<div class="box green"><div class="t">MLflow registry</div><div class="d">LightGBM pyfunc carrying its own feature code, resolved by alias <span class="k">@production</span>. <span class="k">MODEL_URI</span> loads an artifact with no server running.</div><div class="k">ml/</div></div>
|
|
||||||
<div class="box amber"><div class="t">Nextflow DSL2 pipeline</div><div class="d">bcftools normalise → Ensembl VEP → loader. Executors: local, Google Batch, Argo Workflows.</div><div class="k">pipeline/</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
|
|
||||||
<div style="height:16px"></div>
|
|
||||||
|
|
||||||
<div class="row">
|
|
||||||
<div class="band" style="flex:1">
|
|
||||||
<div class="lbl">Deployment track A — serverless (default, ~£1/month idle)</div>
|
|
||||||
<div class="row">
|
|
||||||
<div class="box" style="flex:1"><div class="t">Cloud Run</div><div class="d">api + web, min-instances 0</div></div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">Cloud Run job</div><div class="d">Nextflow driver, started per run</div></div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">Google Batch</div><div class="d">VEP tasks on Spot VMs</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div style="width:14px"></div>
|
|
||||||
<div class="band" style="flex:1">
|
|
||||||
<div class="lbl">Deployment track B — Kubernetes (behind a Terraform flag)</div>
|
|
||||||
<div class="row">
|
|
||||||
<div class="box" style="flex:1"><div class="t">Pub/Sub</div><div class="d">vcf-uploaded topic</div></div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">Argo Events</div><div class="d">sensor → workflow trigger</div></div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">Argo Workflows</div><div class="d">on GKE Autopilot</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<p class="note">Both tracks run the identical pipeline code; the executor is a Nextflow profile, and ArgoCD reconciles the Kubernetes track from the same repository.</p>
|
|
||||||
</div>
|
|
||||||
@@ -1,37 +0,0 @@
|
|||||||
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
||||||
<style>
|
|
||||||
.bar { height:19px; border-radius:3px; background:var(--plum); }
|
|
||||||
.bar.pes { background:var(--plum-soft); border:1px solid var(--plum); }
|
|
||||||
.bar.base { background:var(--soft); }
|
|
||||||
.brow { display:grid; grid-template-columns:210px 1fr 62px; align-items:center; gap:11px; margin-bottom:7px; font-size:12.5px; }
|
|
||||||
.brow .lab { color:var(--soft); text-align:right; }
|
|
||||||
.brow .val { font-family:var(--mono); font-size:11.5px; }
|
|
||||||
.track { background:var(--paper); border-radius:3px; }
|
|
||||||
</style>
|
|
||||||
<div id="frame">
|
|
||||||
<h1>Does the phenotype ranking actually work?</h1>
|
|
||||||
<p class="sub">Every case in Monarch's Phenopacket Store: given a real patient's reported terms, where does the gene their authors diagnosed rank among all 5,269 HPO-annotated genes? Ties give a range — optimistic counts a tie as a win, pessimistic counts every tied gene as ahead.</p>
|
|
||||||
|
|
||||||
<div style="font-size:12px;color:var(--soft);font-weight:600;text-transform:uppercase;letter-spacing:0.08em;margin:4px 0 9px">Causal gene ranked first (top-1), 10,178 published cases</div>
|
|
||||||
<div class="brow"><div class="lab">optimistic</div><div class="track"><div class="bar" style="width:81.0%"></div></div><div class="val">81.0%</div></div>
|
|
||||||
<div class="brow"><div class="lab">pessimistic</div><div class="track"><div class="bar pes" style="width:45.9%"></div></div><div class="val">45.9%</div></div>
|
|
||||||
<div class="brow"><div class="lab">random baseline</div><div class="track"><div class="bar base" style="width:0.4%"></div></div><div class="val">0.02%</div></div>
|
|
||||||
|
|
||||||
<div style="font-size:12px;color:var(--soft);font-weight:600;text-transform:uppercase;letter-spacing:0.08em;margin:20px 0 9px">Causal gene in the top ten</div>
|
|
||||||
<div class="brow"><div class="lab">optimistic</div><div class="track"><div class="bar" style="width:87.2%"></div></div><div class="val">87.2%</div></div>
|
|
||||||
<div class="brow"><div class="lab">pessimistic</div><div class="track"><div class="bar pes" style="width:69.7%"></div></div><div class="val">69.7%</div></div>
|
|
||||||
|
|
||||||
<div style="height:22px"></div>
|
|
||||||
<div class="row">
|
|
||||||
<div class="box amber" style="flex:1"><div class="t">The benchmark is contaminated, and it must be said out loud</div><div class="d">The median causal gene already carries <em>every one</em> of its patient's terms, because HPO's gene annotations are curated from these same case reports. This measures how well the ranking retrieves a gene HPO has already been told about: an upper bound. A prospective number, on a patient whose gene nobody has annotated yet, would be lower, and this corpus cannot say by how much.</div></div>
|
|
||||||
<div style="width:15px"></div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">Measuring my own changes, including the one that failed</div><div class="d">Information-content weighting and ontology propagation both replaced plain term counting. Asked whether they helped, the corpus said only one of them did — pessimistic figures, 6,485 cases with six or more terms:</div>
|
|
||||||
<table style="width:100%;border-collapse:collapse;font-size:11.5px;margin-top:9px">
|
|
||||||
<tr style="color:var(--soft)"><td style="padding:3px 0">count terms (original)</td><td style="text-align:right;font-family:var(--mono)">61.8%</td><td style="text-align:right;font-family:var(--mono)">0.682</td></tr>
|
|
||||||
<tr style="color:var(--green);font-weight:600"><td style="padding:3px 0">+ information content</td><td style="text-align:right;font-family:var(--mono)">63.6%</td><td style="text-align:right;font-family:var(--mono)">0.706</td></tr>
|
|
||||||
<tr style="color:var(--soft)"><td style="padding:3px 0">+ propagation</td><td style="text-align:right;font-family:var(--mono)">58.2%</td><td style="text-align:right;font-family:var(--mono)">0.653</td></tr>
|
|
||||||
<tr><td style="padding:3px 0">+ both (shipped)</td><td style="text-align:right;font-family:var(--mono)">59.5%</td><td style="text-align:right;font-family:var(--mono)">0.670</td></tr>
|
|
||||||
</table>
|
|
||||||
<div class="d" style="margin-top:7px">Weighting earns its place. Propagation costs about what weighting gains — kept for a reason the documentation argues rather than assumes, with the table there so a reader can disagree.</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
@@ -1,22 +0,0 @@
|
|||||||
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
||||||
<div id="frame">
|
|
||||||
<h1>One API call, three execution backends</h1>
|
|
||||||
<p class="sub"><span class="k" style="font-family:var(--mono)">POST /cases/{id}/annotate</span> creates a job row, then <span class="k" style="font-family:var(--mono)">events.launch()</span> picks a backend from configuration alone. The pipeline code never changes.</p>
|
|
||||||
|
|
||||||
<div class="row" style="align-items:flex-start">
|
|
||||||
<div class="box ink" style="flex:0 0 210px"><div class="t">events.launch(job)</div><div class="d">Chooses on settings, in order. Everything downstream writes back to the same <span class="k" style="font-family:var(--mono)">jobs</span> row, so the UI polls one endpoint whichever path ran.</div><div class="k">app/services/events.py</div></div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="col" style="flex:1">
|
|
||||||
<div class="box amber"><div class="t">1. cloudrun_job set → Cloud Run job</div><div class="d">Executes the Nextflow driver as a job with argument overrides, via the Jobs API. Scales to zero between runs; the service account holds only <span class="k" style="font-family:var(--mono)">run.jobsExecutorWithOverrides</span> on that one job.</div></div>
|
|
||||||
<div class="box green"><div class="t">2. pubsub_topic set → Pub/Sub publish</div><div class="d">Publishes the job as an event. Argo Events' sensor subscribes and triggers an Argo Workflow on GKE. Decouples the API from the compute entirely; retries and back-pressure belong to the queue.</div></div>
|
|
||||||
<div class="box plum"><div class="t">3. otherwise → local subprocess</div><div class="d">Runs Nextflow directly and streams stdout into the job log, so the UI can show live progress. This is what a developer gets with no cloud configured at all.</div></div>
|
|
||||||
</div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="col" style="flex:0 0 215px">
|
|
||||||
<div class="box"><div class="t">Nextflow pipeline</div><div class="d">NORMALISE → VEP → LOAD_DB</div></div>
|
|
||||||
<div class="arrow down">↓</div>
|
|
||||||
<div class="box"><div class="t">loader writes results</div><div class="d">Variants inserted, job marked succeeded, and the run records <em>what evidence it looked up</em>.</div><div class="k">has_frequencies, has_effect_scores</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<p class="note">The database URL never appears on a command line: it is passed by environment, or as a Nextflow secret, so it stays out of <span class="k" style="font-family:var(--mono)">.command.sh</span> and the workflow logs.</p>
|
|
||||||
</div>
|
|
||||||
@@ -1,30 +0,0 @@
|
|||||||
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
||||||
<div id="frame">
|
|
||||||
<h1>Evidence that was never looked up must abstain</h1>
|
|
||||||
<p class="sub">The rank is a weighted mean of four auditable components. Which ones may score is decided per job, from what the annotation run actually produced.</p>
|
|
||||||
<div class="row">
|
|
||||||
<div class="box plum" style="flex:1"><div class="t">phenotype — 0.35</div><div class="d">Share of the patient's HPO terms annotated to the gene, each weighted by its information content.</div><div class="k">always scores</div></div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">rarity — 0.25</div><div class="d">Allele frequency in gnomAD, as a step function a reviewer can read.</div><div class="k">only if the run looked up frequencies</div></div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">consequence — 0.20</div><div class="d">Predicted impact severity: HIGH, MODERATE, LOW, MODIFIER.</div><div class="k">always scores</div></div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">model — 0.20</div><div class="d">LightGBM P(pathogenic) from variant effect.</div><div class="k">only with CADD or AlphaMissense</div></div>
|
|
||||||
</div>
|
|
||||||
<div style="height:20px"></div>
|
|
||||||
<div class="row">
|
|
||||||
<div class="band" style="flex:1">
|
|
||||||
<div class="lbl">Before — the bug</div>
|
|
||||||
<div class="box" style="border-color:var(--plum)"><div class="t" style="color:var(--plum)">rarity_score(None) → 1.00</div><div class="d">"No frequency in this run" was read as "absent from gnomAD, therefore maximally rare", and every variant collected a free 0.25. The model, meanwhile, scored 0.887 on features it had never seen. Two of four components were fiction, and the score looked fully informed.</div></div>
|
|
||||||
</div>
|
|
||||||
<div style="width:16px"></div>
|
|
||||||
<div class="band" style="flex:1">
|
|
||||||
<div class="lbl">After — abstention and renormalisation</div>
|
|
||||||
<div class="box" style="border-color:var(--green)"><div class="t" style="color:var(--green)">component → null, weights renormalised</div><div class="d">The job records <span class="k" style="font-family:var(--mono)">has_frequencies</span> and <span class="k" style="font-family:var(--mono)">has_effect_scores</span>. Absent components drop out and the remaining weights renormalise over 0.55, so the score stays on a 0–1 scale and still means the same thing. The UI prints "not looked up", never a bar.</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
<div style="height:18px"></div>
|
|
||||||
<table style="width:100%; border-collapse:collapse; font-size:12.5px; border:1px solid var(--line)">
|
|
||||||
<tr style="background:var(--paper)"><th style="text-align:left;padding:7px 10px;border-bottom:1px solid var(--line)">Published Loeys–Dietz case, run without a VEP cache</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">score</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">phenotype (0.64)</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">rarity</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">consequence (0.36)</th><th style="padding:7px 10px;border-bottom:1px solid var(--line)">model</th></tr>
|
|
||||||
<tr><td style="padding:7px 10px;border-bottom:1px solid var(--line)"><em>TGFBR2</em> 3:30672252 G>T missense — the published diagnosis</td><td style="padding:7px 10px;text-align:center;border-bottom:1px solid var(--line)"><strong style="color:var(--plum)">0.855</strong></td><td style="padding:7px 10px;text-align:center;border-bottom:1px solid var(--line)">1.00 <span style="color:var(--soft)">(30/30)</span></td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic;border-bottom:1px solid var(--line)">not looked up</td><td style="padding:7px 10px;text-align:center;border-bottom:1px solid var(--line)">0.60</td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic;border-bottom:1px solid var(--line)">not looked up</td></tr>
|
|
||||||
<tr><td style="padding:7px 10px"><em>OSBPL10</em> 3:31748090 missense — incidental</td><td style="padding:7px 10px;text-align:center">0.218</td><td style="padding:7px 10px;text-align:center">0.00</td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic">not looked up</td><td style="padding:7px 10px;text-align:center">0.60</td><td style="padding:7px 10px;text-align:center;color:var(--soft);font-style:italic">not looked up</td></tr>
|
|
||||||
</table>
|
|
||||||
<p class="note">Both are rare missense variants, identical on every piece of evidence this run holds except one. The phenotype is what separates a published diagnosis from an incidental variant in a lipid-transport gene — which is the argument for phenotype-driven triage, in one table.</p>
|
|
||||||
</div>
|
|
||||||
@@ -1,25 +0,0 @@
|
|||||||
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
||||||
<style>
|
|
||||||
.big { font-family:var(--mono); font-size:34px; font-weight:600; letter-spacing:-0.02em; }
|
|
||||||
.mrow { display:grid; grid-template-columns:1fr 92px 92px 110px 110px; font-size:13px; align-items:center; }
|
|
||||||
.mrow > div { padding:8px 10px; border-bottom:1px solid var(--line); }
|
|
||||||
.mhead > div { color:var(--soft); font-weight:600; font-size:11px; text-transform:uppercase; letter-spacing:0.06em; background:var(--paper); }
|
|
||||||
.num { font-family:var(--mono); text-align:right; }
|
|
||||||
</style>
|
|
||||||
<div id="frame">
|
|
||||||
<h1>The number that changed what the model is allowed to do</h1>
|
|
||||||
<p class="sub">Held-out evaluation with whole genes held out, never single variants (Grimm <em>et al.</em> 2015): 312,025 training and 74,239 test variants across 7,728 and 1,932 genes, with no gene on both sides.</p>
|
|
||||||
<div class="mrow mhead"><div>Model</div><div class="num">AUROC</div><div class="num">AUPRC</div><div class="num">missense AUROC</div><div class="num">missense AUPRC</div></div>
|
|
||||||
<div class="mrow"><div>v2 — with gnomAD allele frequency as a feature</div><div class="num">0.986</div><div class="num">0.954</div><div class="num">0.872</div><div class="num">0.725</div></div>
|
|
||||||
<div class="mrow"><div>v3 — allele frequency removed</div><div class="num">0.966</div><div class="num">0.881</div><div class="num" style="color:var(--plum);font-weight:700">0.500</div><div class="num">0.398</div></div>
|
|
||||||
<div style="height:20px"></div>
|
|
||||||
<div class="row">
|
|
||||||
<div class="box plum" style="flex:0 0 268px; text-align:center; padding:17px"><div class="big" style="color:var(--plum)">0.500</div><div class="d" style="margin-top:5px">AUROC on missense variants once frequency is removed. Exactly random.</div></div>
|
|
||||||
<div style="width:15px"></div>
|
|
||||||
<div class="col" style="flex:1">
|
|
||||||
<div class="box"><div class="t">What that proves</div><div class="d">Strip frequency out and the model cannot tell one missense variant from another at all — nothing is left but the consequence class, so every missense row scores identically. The respectable-looking 0.872 was never variant-effect knowledge. It was allele frequency.</div></div>
|
|
||||||
<div class="box amber"><div class="t">And the frequency feature was circular</div><div class="d">ACMG's BA1/BS1 criteria assign ClinVar's <em>benign</em> labels <em>using</em> allele frequency. The feature had partly caused the label, so the model was rediscovering the rule that produced its own training data.</div></div>
|
|
||||||
<div class="box green"><div class="t">The consequence for the product</div><div class="d">Frequency is no longer a feature — the ranking already scores it explicitly and auditably, and feeding it to the model as well put ~45% of every rank on one measurement counted twice. The model now abstains unless it has CADD or AlphaMissense, because 0.500 is the measurement saying it has nothing else to add.</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
@@ -1,18 +0,0 @@
|
|||||||
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
||||||
<div id="frame">
|
|
||||||
<h1>The Nextflow DSL2 pipeline</h1>
|
|
||||||
<p class="sub">Three processes, each a container. The same workflow file runs on a laptop, on Google Batch and under Argo Workflows — the executor is a profile, not a rewrite.</p>
|
|
||||||
<div class="row" style="align-items:stretch">
|
|
||||||
<div class="box" style="flex:1"><div class="t">NORMALISE</div><div class="d">bcftools norm: left-align and split multi-allelics, then set the VCF ID to <span class="k" style="font-family:var(--mono)">CHROM_POS_REF_ALT</span>.</div><div class="k">bcftools 1.20</div></div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="box amber" style="flex:1.25"><div class="t">VEP</div><div class="d">Ensembl VEP 113. Cache mode for full annotation, or <span class="k" style="font-family:var(--mono)">--database</span> against Ensembl's public server when the 25 GB cache is not available. CADD and AlphaMissense are optional plugins.</div><div class="k">4 cpus, 8 GB</div></div>
|
|
||||||
<div class="arrow">→</div>
|
|
||||||
<div class="box" style="flex:1"><div class="t">LOAD_DB</div><div class="d">Parses VEP <span class="k" style="font-family:var(--mono)">--tab</span> output, inserts variants idempotently (a retried task cannot duplicate), and marks the job succeeded.</div><div class="k">psycopg, chunked COPY</div></div>
|
|
||||||
</div>
|
|
||||||
<div style="height:18px"></div>
|
|
||||||
<div class="row">
|
|
||||||
<div class="box plum" style="flex:1"><div class="t">Why the ID carries identity</div><div class="d">VEP's own Location and Allele columns trim indel alleles and shift positions, so a deletion round-trips as the wrong variant. Carrying <span class="k" style="font-family:var(--mono)">CHROM_POS_REF_ALT</span> through the ID field keeps the exact alleles the caller emitted. Verified on a real run: <span class="k" style="font-family:var(--mono)">22:42126611 CT>C</span> survives intact.</div></div>
|
|
||||||
<div style="width:14px"></div>
|
|
||||||
<div class="box green" style="flex:1"><div class="t">Stub blocks keep CI honest</div><div class="d">Every process has a <span class="k" style="font-family:var(--mono)">stub:</span> block, so CI runs <span class="k" style="font-family:var(--mono)">nextflow run main.nf -stub-run</span> on a three-record fixture and checks wiring and channel shapes on every pull request — no containers, no VEP cache, seconds not hours.</div></div>
|
|
||||||
</div>
|
|
||||||
</div>
|
|
||||||
@@ -1,67 +0,0 @@
|
|||||||
import { chromium } from 'playwright';
|
|
||||||
import { mkdirSync } from 'node:fs';
|
|
||||||
import { resolve } from 'node:path';
|
|
||||||
|
|
||||||
const BASE = process.env.BASE ?? 'http://localhost:5173';
|
|
||||||
const CASE = process.env.CASE_ID ?? '91d85d3d-3351-4a7d-8d29-164cdea76518';
|
|
||||||
const OUT = resolve('out');
|
|
||||||
mkdirSync(OUT, { recursive: true });
|
|
||||||
|
|
||||||
const shot = async (page, name, opts = {}) => {
|
|
||||||
await page.screenshot({ path: `${OUT}/${name}.png`, ...opts });
|
|
||||||
console.log('wrote', name);
|
|
||||||
};
|
|
||||||
|
|
||||||
const EXE = process.env.CHROME_PATH;
|
|
||||||
const browser = await chromium.launch(EXE ? { executablePath: EXE } : {});
|
|
||||||
const page = await browser.newPage({
|
|
||||||
viewport: { width: 1440, height: 1000 },
|
|
||||||
deviceScaleFactor: 2
|
|
||||||
});
|
|
||||||
|
|
||||||
// 1. case list
|
|
||||||
await page.goto(`${BASE}/`, { waitUntil: 'networkidle' });
|
|
||||||
await page.waitForSelector('.caselist li', { timeout: 20000 });
|
|
||||||
await shot(page, '01-cases', { fullPage: true });
|
|
||||||
|
|
||||||
// 2. the published case: funnel, filters, ranked candidates
|
|
||||||
await page.goto(`${BASE}/cases/${CASE}`, { waitUntil: 'networkidle' });
|
|
||||||
await page.waitForSelector('.candidate', { timeout: 30000 });
|
|
||||||
await page.waitForTimeout(600);
|
|
||||||
await shot(page, '02-case', { fullPage: true });
|
|
||||||
|
|
||||||
// 3. the evidence panel for the top candidate: the components table, abstentions and all
|
|
||||||
await page.locator('.candidate').first().click();
|
|
||||||
await page.waitForSelector('.panel', { timeout: 20000 });
|
|
||||||
await page.waitForTimeout(600);
|
|
||||||
await shot(page, '03-variant-panel', { fullPage: true });
|
|
||||||
|
|
||||||
// just the panel, cropped, for a tighter figure
|
|
||||||
const panel = page.locator('.panel').first();
|
|
||||||
await panel.screenshot({ path: `${OUT}/04-panel-only.png` });
|
|
||||||
console.log('wrote 04-panel-only');
|
|
||||||
|
|
||||||
// just the funnel
|
|
||||||
const funnel = page.locator('.funnel, [class*="funnel"]').first();
|
|
||||||
if (await funnel.count()) {
|
|
||||||
await funnel.screenshot({ path: `${OUT}/05-funnel.png` });
|
|
||||||
console.log('wrote 05-funnel');
|
|
||||||
}
|
|
||||||
|
|
||||||
// 4. the case report
|
|
||||||
await page.goto(`${BASE}/cases/${CASE}/report`, { waitUntil: 'networkidle' });
|
|
||||||
await page.waitForTimeout(1200);
|
|
||||||
await shot(page, '06-report', { fullPage: true });
|
|
||||||
|
|
||||||
// 5. local diagram pages rendered to PNG
|
|
||||||
for (const name of process.env.DIAGRAMS?.split(',').filter(Boolean) ?? []) {
|
|
||||||
await page.goto(`file://${resolve(`${name}.html`)}`, {
|
|
||||||
waitUntil: 'networkidle'
|
|
||||||
});
|
|
||||||
await page.waitForTimeout(300);
|
|
||||||
const box = page.locator('#frame');
|
|
||||||
await box.screenshot({ path: `${OUT}/${name}.png` });
|
|
||||||
console.log('wrote', name);
|
|
||||||
}
|
|
||||||
|
|
||||||
await browser.close();
|
|
||||||
|
Before Width: | Height: | Size: 180 KiB |
|
Before Width: | Height: | Size: 301 KiB |
|
Before Width: | Height: | Size: 611 KiB |
|
Before Width: | Height: | Size: 276 KiB |
|
Before Width: | Height: | Size: 26 KiB |
|
Before Width: | Height: | Size: 340 KiB |
|
Before Width: | Height: | Size: 279 KiB |
|
Before Width: | Height: | Size: 254 KiB |
|
Before Width: | Height: | Size: 233 KiB |
|
Before Width: | Height: | Size: 287 KiB |
|
Before Width: | Height: | Size: 222 KiB |
|
Before Width: | Height: | Size: 188 KiB |
@@ -1,382 +0,0 @@
|
|||||||
% rarelens - a technical introduction
|
|
||||||
% Build: make -C docs/blog (or: pdflatex rarelens.tex, twice)
|
|
||||||
\documentclass[11pt,a4paper]{article}
|
|
||||||
|
|
||||||
\usepackage[T1]{fontenc}
|
|
||||||
\usepackage[utf8]{inputenc}
|
|
||||||
\usepackage{lmodern}
|
|
||||||
\usepackage{microtype}
|
|
||||||
\usepackage[margin=2.4cm,top=2.6cm,bottom=2.6cm]{geometry}
|
|
||||||
\usepackage{graphicx}
|
|
||||||
\usepackage{booktabs}
|
|
||||||
\usepackage{xcolor}
|
|
||||||
\usepackage{caption}
|
|
||||||
\usepackage{enumitem}
|
|
||||||
\usepackage{fancyhdr}
|
|
||||||
\usepackage{titlesec}
|
|
||||||
\usepackage{listings}
|
|
||||||
\usepackage{float}
|
|
||||||
\usepackage[hidelinks]{hyperref}
|
|
||||||
|
|
||||||
\definecolor{ink}{HTML}{16243A}
|
|
||||||
\definecolor{soft}{HTML}{52627A}
|
|
||||||
\definecolor{plum}{HTML}{7A1F5C}
|
|
||||||
\definecolor{line}{HTML}{CFD6DF}
|
|
||||||
\definecolor{paper}{HTML}{F3F5F7}
|
|
||||||
|
|
||||||
\hypersetup{colorlinks=true,linkcolor=plum,urlcolor=plum,citecolor=plum}
|
|
||||||
\color{ink}
|
|
||||||
|
|
||||||
\titleformat{\section}{\normalfont\Large\bfseries\color{ink}}{\thesection}{0.7em}{}
|
|
||||||
\titleformat{\subsection}{\normalfont\large\bfseries\color{ink}}{\thesubsection}{0.6em}{}
|
|
||||||
\titlespacing*{\section}{0pt}{1.6em}{0.6em}
|
|
||||||
|
|
||||||
\captionsetup{font=small,labelfont={bf,color=soft},textfont={color=soft},width=0.94\textwidth}
|
|
||||||
\setlist[itemize]{leftmargin=1.2em,itemsep=0.25em,topsep=0.35em}
|
|
||||||
\setlength{\parskip}{0.55em}
|
|
||||||
\setlength{\parindent}{0pt}
|
|
||||||
|
|
||||||
\pagestyle{fancy}
|
|
||||||
\fancyhf{}
|
|
||||||
\fancyhead[L]{\small\color{soft}rarelens --- a technical introduction}
|
|
||||||
\fancyfoot[C]{\small\color{soft}\thepage}
|
|
||||||
\renewcommand{\headrulewidth}{0.4pt}
|
|
||||||
|
|
||||||
\lstdefinestyle{code}{
|
|
||||||
basicstyle=\ttfamily\footnotesize\color{ink},
|
|
||||||
backgroundcolor=\color{paper},
|
|
||||||
frame=single, rulecolor=\color{line}, framesep=5pt,
|
|
||||||
breaklines=true, showstringspaces=false,
|
|
||||||
keywordstyle=\color{plum}, commentstyle=\color{soft}\itshape,
|
|
||||||
xleftmargin=0pt, aboveskip=0.9em, belowskip=0.9em
|
|
||||||
}
|
|
||||||
\lstset{style=code}
|
|
||||||
|
|
||||||
% LaTeX is conservative about letting text share a page with a large float, which on a document
|
|
||||||
% this figure-heavy strands them on near-empty pages of their own. Loosen the thresholds.
|
|
||||||
\renewcommand{\topfraction}{0.92}
|
|
||||||
\renewcommand{\bottomfraction}{0.80}
|
|
||||||
\renewcommand{\textfraction}{0.06}
|
|
||||||
\renewcommand{\floatpagefraction}{0.82}
|
|
||||||
\setcounter{topnumber}{2}
|
|
||||||
\setcounter{totalnumber}{3}
|
|
||||||
|
|
||||||
% htbp, not H: a full-width figure that will not fit pinned in place leaves half a page blank.
|
|
||||||
\newcommand{\fig}[3]{%
|
|
||||||
\begin{figure}[htbp]\centering
|
|
||||||
\includegraphics[width=#2\textwidth]{#1}
|
|
||||||
\caption{#3}\end{figure}}
|
|
||||||
|
|
||||||
\begin{document}
|
|
||||||
|
|
||||||
\begin{center}
|
|
||||||
{\LARGE\bfseries rarelens}\\[0.35em]
|
|
||||||
{\large\color{soft} An end-to-end rare-disease variant triage platform}\\[1.1em]
|
|
||||||
{\color{soft}\small A technical introduction --- pipeline, infrastructure, model and the evidence behind the ranking}
|
|
||||||
\end{center}
|
|
||||||
|
|
||||||
\vspace{0.6em}
|
|
||||||
\hrule
|
|
||||||
\vspace{1.2em}
|
|
||||||
|
|
||||||
\section*{In one paragraph}
|
|
||||||
|
|
||||||
\textbf{rarelens takes a patient's genome and their clinical phenotype and narrows thousands of
|
|
||||||
variants to a handful a scientist can actually review, showing the evidence behind every rank.}
|
|
||||||
It is a working monorepo: a Nextflow DSL2 pipeline that annotates variants with Ensembl VEP, a
|
|
||||||
FastAPI service that ranks them against Human Phenotype Ontology annotations, a LightGBM model
|
|
||||||
served from an MLflow registry, a SvelteKit interface built around triage decisions rather than
|
|
||||||
table filtering, and Terraform for two deployment tracks on Google Cloud --- a serverless one that
|
|
||||||
idles at roughly \pounds1 a month, and a Kubernetes one with Argo Workflows, Argo Events and
|
|
||||||
ArgoCD behind a feature flag. Everything runs on public, openly licensed data. No patient data is
|
|
||||||
used, accepted, or possible to load.
|
|
||||||
|
|
||||||
This document is a technical tour. It is also, deliberately, an account of the things that turned
|
|
||||||
out to be wrong --- including a model metric that collapsed from 0.872 to 0.500 the moment I took
|
|
||||||
away the feature that was doing all the work, and an ontology bug that silently deleted 399 terms.
|
|
||||||
|
|
||||||
\section{The problem}
|
|
||||||
|
|
||||||
A rare-disease proband's exome contains something like twenty thousand coding variants. Perhaps a
|
|
||||||
few hundred are rare and protein-altering. Exactly one, usually, explains why the patient is ill.
|
|
||||||
Finding it is not a filtering problem so much as an evidence problem: the same variant is
|
|
||||||
uninteresting in one patient and diagnostic in another, and what changes between them is the
|
|
||||||
clinical phenotype.
|
|
||||||
|
|
||||||
That observation drives the whole design. The interface is not a variant table with filters --- it
|
|
||||||
is a ranked shortlist where each candidate carries the four pieces of evidence that put it there,
|
|
||||||
and where a reviewer shortlists or dismisses with a reason that ends up in a case report.
|
|
||||||
|
|
||||||
\fig{figures/arch.png}{1.0}{The system. One repository holds the pipeline, the API, the interface,
|
|
||||||
model serving and the infrastructure for two deployment tracks. The scientist never sees any of it
|
|
||||||
except the middle column.}
|
|
||||||
|
|
||||||
\section{The Nextflow pipeline}
|
|
||||||
|
|
||||||
Annotation is a three-process DSL2 workflow, each process a pinned container: \texttt{bcftools}
|
|
||||||
normalises, Ensembl VEP 113 annotates, and a loader writes results into PostgreSQL and marks the
|
|
||||||
job succeeded.
|
|
||||||
|
|
||||||
\fig{figures/pipeline.png}{1.0}{The pipeline. The same workflow file runs on a laptop, on Google
|
|
||||||
Batch and under Argo Workflows; the executor is a profile, not a rewrite.}
|
|
||||||
|
|
||||||
Two details are worth pulling out, because both cost real debugging time.
|
|
||||||
|
|
||||||
\subsection{Variant identity survives annotation}
|
|
||||||
|
|
||||||
VEP's \texttt{Location} and \texttt{Allele} output columns trim indel alleles and shift positions
|
|
||||||
by one, which is correct for VEP's own purposes and wrong as a primary key. A deletion written by
|
|
||||||
the caller as \texttt{CT>C} comes back as \texttt{->} at a different coordinate, and the loader
|
|
||||||
then stores a variant that does not exist in the input file.
|
|
||||||
|
|
||||||
The fix is to carry identity through a field VEP does not touch. \texttt{NORMALISE} sets the VCF ID
|
|
||||||
to \texttt{CHROM\_POS\_REF\_ALT}, and the loader parses identity back out of it:
|
|
||||||
|
|
||||||
\begin{lstlisting}[language=bash]
|
|
||||||
bcftools norm -m -any -f $genome | \
|
|
||||||
bcftools annotate --set-id '%CHROM\_%POS\_%REF\_%FIRST_ALT'
|
|
||||||
\end{lstlisting}
|
|
||||||
|
|
||||||
Verified on a real run: \texttt{22:42126611 CT>C} round-trips with its alleles intact.
|
|
||||||
|
|
||||||
\subsection{Stub blocks make continuous integration possible at all}
|
|
||||||
|
|
||||||
A VEP cache is 25\,GB. No pull request is going to download one. Every process therefore carries a
|
|
||||||
\texttt{stub:} block, and CI runs the workflow with \texttt{-stub-run} against a three-record
|
|
||||||
fixture, checking channel wiring and process contracts in seconds without a container or a cache in
|
|
||||||
sight. It catches the failure that actually happens in practice --- a renamed output, a channel
|
|
||||||
that emits the wrong cardinality --- while leaving the scientific correctness to the tests that can
|
|
||||||
afford to be slow.
|
|
||||||
|
|
||||||
\section{Event-driven execution, three ways}
|
|
||||||
|
|
||||||
\texttt{POST /cases/\{id\}/annotate} writes a job row and hands off. What happens next is a
|
|
||||||
configuration decision, not a code path the caller knows about.
|
|
||||||
|
|
||||||
\fig{figures/events.png}{1.0}{One entry point, three backends, chosen from settings alone.}
|
|
||||||
|
|
||||||
\begin{itemize}
|
|
||||||
\item \textbf{Cloud Run job.} The Nextflow driver runs as a Cloud Run job started through the
|
|
||||||
Jobs API with argument overrides. It scales to zero between runs, and its service account
|
|
||||||
holds \texttt{run.jobsExecutorWithOverrides} on exactly one job --- not project-wide.
|
|
||||||
\item \textbf{Pub/Sub.} The job is published as an event. An Argo Events sensor subscribes and
|
|
||||||
triggers an Argo Workflow on GKE. This is the decoupled path: retries, ordering and
|
|
||||||
back-pressure become the queue's problem rather than the API's, and the API can be
|
|
||||||
restarted mid-pipeline without losing work.
|
|
||||||
\item \textbf{Local subprocess.} Nextflow runs directly and its stdout is streamed into the job
|
|
||||||
log, so the interface shows live progress. This is what a developer gets with nothing
|
|
||||||
configured, and it is the same code path the other two wrap.
|
|
||||||
\end{itemize}
|
|
||||||
|
|
||||||
All three converge on the same \texttt{jobs} row, so the interface polls one endpoint regardless.
|
|
||||||
The database URL is passed by environment or as a Nextflow secret and never appears on a command
|
|
||||||
line, keeping it out of \texttt{.command.sh} and the workflow logs.
|
|
||||||
|
|
||||||
\section{Kubernetes, Argo and GitOps}
|
|
||||||
|
|
||||||
The Kubernetes track is Kustomize bases with a local overlay (kind, an in-cluster Postgres) and a
|
|
||||||
GCP overlay (Cloud SQL, Workload Identity). Argo Workflows runs the annotation
|
|
||||||
\texttt{WorkflowTemplate}; Argo Events holds the Pub/Sub EventSource and the sensor that triggers
|
|
||||||
it; ArgoCD reconciles the cluster from the repository, and a green CI run on \texttt{main} bumps
|
|
||||||
image tags in the GCP overlay so that deployment is a commit rather than a command.
|
|
||||||
|
|
||||||
One bug from this area is worth recording because it is invisible until it bites: Kustomize
|
|
||||||
generates hashed ConfigMap names so that a configuration change forces a rollout, but the hashed
|
|
||||||
name is only substituted into workloads Kustomize believes are in scope. The overlays were missing
|
|
||||||
\texttt{namespace: rarelens}, so the substitution silently did not happen and pods mounted a
|
|
||||||
ConfigMap name that no longer existed. The symptom was a pod stuck in \texttt{CreateContainerConfigError}
|
|
||||||
with nothing wrong in the manifests as written.
|
|
||||||
|
|
||||||
\section{Infrastructure as code}
|
|
||||||
|
|
||||||
Terraform provisions both tracks from one root module, with the expensive half behind flags:
|
|
||||||
|
|
||||||
\begin{lstlisting}[language=bash]
|
|
||||||
terraform apply -var project=<id> # serverless: Cloud Run + Batch
|
|
||||||
terraform apply -var project=<id> -var deploy_kubernetes=true \
|
|
||||||
-var deploy_cloud_sql=true # adds GKE, Argo, Cloud SQL
|
|
||||||
\end{lstlisting}
|
|
||||||
|
|
||||||
The default track provisions Cloud Run services for the API and interface, a Cloud Run job for the
|
|
||||||
Nextflow driver, Google Batch for pipeline tasks on Spot VMs, a GCS bucket, Secret Manager entries
|
|
||||||
and Artifact Registry. GKE Autopilot and Cloud SQL are opt-in, because a Kubernetes control plane
|
|
||||||
and a managed database are most of what a demonstration estate costs.
|
|
||||||
|
|
||||||
CI authenticates to Google Cloud through Workload Identity Federation, so there is no service
|
|
||||||
account key anywhere in the repository or in GitHub secrets. Terraform is validated and
|
|
||||||
format-checked on every pull request.
|
|
||||||
|
|
||||||
\subsection{Cost as a design constraint}
|
|
||||||
|
|
||||||
A portfolio platform is idle more than 99\% of the time, which makes idle cost the only cost that
|
|
||||||
matters. The serverless track is built around that: Cloud Run at \texttt{min-instances=0}, a driver
|
|
||||||
that exists only while a pipeline runs, and Batch on Spot. Idle cost lands near \pounds1 a month,
|
|
||||||
almost all of it the database. The Kubernetes track exists to demonstrate the GitOps path and is
|
|
||||||
meant to be destroyed afterwards.
|
|
||||||
|
|
||||||
\section{Integrating external systems}
|
|
||||||
|
|
||||||
Almost nothing here is self-contained, and integrating public biological infrastructure is most of
|
|
||||||
the work:
|
|
||||||
|
|
||||||
\begin{itemize}
|
|
||||||
\item \textbf{Ensembl VEP} in cache mode, or against Ensembl's \emph{public database server}
|
|
||||||
with \texttt{--database} when 25\,GB is not available --- slower per variant, no plugin
|
|
||||||
scores, but no download.
|
|
||||||
\item \textbf{gnomAD v4.1} allele frequencies streamed by genomic region straight out of the
|
|
||||||
public Google Cloud bucket. The files are tabix-indexed, so a range request returns a few
|
|
||||||
hundred records without fetching the file. I had previously written off frequencies as
|
|
||||||
impossible without the full cache; testing that assumption disproved it.
|
|
||||||
\item \textbf{Human Phenotype Ontology} gene-to-phenotype annotations plus the ontology itself,
|
|
||||||
propagated and weighted at load time.
|
|
||||||
\item \textbf{ClinVar} for model training labels, filtered to two-star review status and above.
|
|
||||||
\item \textbf{Phenopacket Store} (Monarch Initiative), which curates published case reports into
|
|
||||||
GA4GH phenopackets --- the source of both the demonstration case and the benchmark.
|
|
||||||
\item \textbf{MLflow} as a model registry, resolved by alias, with a registry-free path for
|
|
||||||
deployments that should not run a tracking server.
|
|
||||||
\end{itemize}
|
|
||||||
|
|
||||||
\begin{lstlisting}[language=bash]
|
|
||||||
# real gnomAD frequencies with no bulk download, verified end to end
|
|
||||||
bcftools view -r chr3:30672000-30673000 \
|
|
||||||
https://storage.googleapis.com/gcp-public-data--gnomad/release/4.1/...chr3.vcf.bgz
|
|
||||||
# -> 392 records, 5 KB, then fed to VEP with --custom alongside --database
|
|
||||||
\end{lstlisting}
|
|
||||||
|
|
||||||
\section{The ranking, and what it refuses to claim}
|
|
||||||
|
|
||||||
The rank is a weighted mean of four components a reviewer can audit. ClinVar is deliberately
|
|
||||||
\emph{not} one of them --- it sits beside the result as independent confirmation, so nothing ranks
|
|
||||||
highly merely because ClinVar already called it pathogenic.
|
|
||||||
|
|
||||||
\fig{figures/evidence.png}{1.0}{Rarity and consequence filter; phenotype only ranks, because a real
|
|
||||||
diagnosis can sit in a gene nobody has annotated yet. A component with no evidence behind it
|
|
||||||
abstains, and the remaining weights renormalise.}
|
|
||||||
|
|
||||||
The abstention rule replaced a genuine bug. Run without a VEP cache, there are no allele
|
|
||||||
frequencies --- and the code read a missing frequency as \emph{absent from gnomAD, therefore
|
|
||||||
maximally rare}, handing every variant a free quarter of its score. The model, separately, was
|
|
||||||
returning 0.887 for every variant on features it had never been given. Two of four components were
|
|
||||||
fiction, and the total looked fully informed.
|
|
||||||
|
|
||||||
The fix is structural rather than cosmetic: the loader records what the annotation run actually
|
|
||||||
produced (\texttt{has\_frequencies}, \texttt{has\_effect\_scores}), components without evidence
|
|
||||||
return null instead of a number, and the weights renormalise over whatever is left. The interface
|
|
||||||
prints ``not looked up'' where it would otherwise have drawn a bar.
|
|
||||||
|
|
||||||
\section{Measuring the ranking}
|
|
||||||
|
|
||||||
One demonstration case ranking correctly is an anecdote. The benchmark asks the only question that
|
|
||||||
matters for a phenotype-driven tool, across every usable case in Phenopacket Store.
|
|
||||||
|
|
||||||
\fig{figures/benchmark.png}{1.0}{Retrieval against 5,269 candidate genes, and an honest report of
|
|
||||||
what the measurement cannot tell you.}
|
|
||||||
|
|
||||||
Two things in that figure matter more than the headline. The first is the contamination: HPO's gene
|
|
||||||
annotations are curated from these same case reports, so the median causal gene already carries
|
|
||||||
every one of its patient's terms. The number is an upper bound and is labelled as one. The second
|
|
||||||
is that when I measured my own improvements, one of them did not work --- information-content
|
|
||||||
weighting helped, ontology propagation cost about as much as weighting gained. That result is in
|
|
||||||
the documentation with the table, rather than quietly dropped.
|
|
||||||
|
|
||||||
\section{The model, and the number that changed it}
|
|
||||||
|
|
||||||
The pathogenicity model is LightGBM trained on ClinVar labels, held out \emph{by gene} rather than
|
|
||||||
by variant. That distinction is not pedantry: a random split puts variants of the same gene on both
|
|
||||||
sides, and the model then scores the gene instead of the variant, which is exactly the inflation
|
|
||||||
Grimm \emph{et al.} documented for this class of tool in 2015.
|
|
||||||
|
|
||||||
\fig{figures/model.png}{1.0}{Removing one feature moved missense AUROC from 0.872 to 0.500.}
|
|
||||||
|
|
||||||
The model originally took allele frequency as a feature and looked respectable. Probing it showed
|
|
||||||
frequency dominating everything --- the same missense variant scored 0.887 at frequency zero and
|
|
||||||
0.0003 at one per cent. That is two separate problems. It double-counted, because the ranking
|
|
||||||
already scores frequency explicitly, putting roughly 45\% of every rank on one measurement. And it
|
|
||||||
was circular, because ACMG's BA1/BS1 criteria assign ClinVar's benign labels \emph{using} allele
|
|
||||||
frequency, so the model was rediscovering the rule that had generated its own labels.
|
|
||||||
|
|
||||||
Retraining without it returned missense AUROC 0.500 --- exactly random. With frequency gone and no
|
|
||||||
CADD or AlphaMissense scores in the training table, nothing is left but the consequence class, so
|
|
||||||
every missense variant scores identically. The conclusion is unambiguous and slightly
|
|
||||||
uncomfortable: the model never had variant-effect knowledge. It now abstains from the ranking
|
|
||||||
unless it has a predictor the other components do not already provide, and 0.500 is the measurement
|
|
||||||
that justifies the abstention.
|
|
||||||
|
|
||||||
\section{The application}
|
|
||||||
|
|
||||||
The interface is built around the narrowing and the decision, not around the table.
|
|
||||||
|
|
||||||
\fig{figures/02-case.png}{1.0}{A published case. The funnel across the top is the story --- variants
|
|
||||||
called, rarity, coding candidates, phenotype-matched --- and each candidate carries its evidence as
|
|
||||||
chips. The note under the header states plainly which components did not score and why.}
|
|
||||||
|
|
||||||
\fig{figures/04-panel-only.png}{0.62}{Selecting a candidate opens the arithmetic. Every component is
|
|
||||||
shown with its weight and its contribution, and the two that had nothing to go on say so rather
|
|
||||||
than displaying a zero.}
|
|
||||||
|
|
||||||
\fig{figures/06-report.png}{0.92}{The case report: shortlisted and dismissed variants with the
|
|
||||||
reviewer's reasons, the funnel counts, and provenance --- VEP version, model version, run time ---
|
|
||||||
so a result can be reproduced or challenged later.}
|
|
||||||
|
|
||||||
\section{Engineering practice}
|
|
||||||
|
|
||||||
\begin{itemize}
|
|
||||||
\item \textbf{Tests where the risk is.} 99 Python tests plus 49 in the front end, concentrated on
|
|
||||||
the ranking arithmetic, the ontology handling, the loader's idempotency and the security
|
|
||||||
boundary. Database tests run against a real PostgreSQL --- an embedded server locally, a
|
|
||||||
service container in CI --- because the schema is part of the behaviour.
|
|
||||||
\item \textbf{Migrations are tested, both ways.} Alembic upgrade and downgrade are exercised
|
|
||||||
against a live database, which is how a downgrade that left a stale enum type behind was
|
|
||||||
caught.
|
|
||||||
\item \textbf{CI runs the whole estate:} lint and types for the API, unit tests for the model
|
|
||||||
code, a real Postgres for the loader, a Nextflow stub run, and \texttt{terraform validate}.
|
|
||||||
\item \textbf{Input validation as a security boundary.} A case's \texttt{vcf\_uri} must be a
|
|
||||||
\texttt{gs://} object or an absolute path beneath a configured data root, with a VCF
|
|
||||||
suffix. That is what stops a crafted path becoming a Nextflow option or reading an
|
|
||||||
arbitrary file, and it has its own test file.
|
|
||||||
\end{itemize}
|
|
||||||
|
|
||||||
\section{Deployment}
|
|
||||||
|
|
||||||
The demonstration runs on Railway --- three services, with the API reachable only on the private
|
|
||||||
network so the interface's \texttt{/api} proxy is the single public entry point. One shared
|
|
||||||
credential sits in front of it.
|
|
||||||
|
|
||||||
Railway cannot run the pipeline: Nextflow shells out to \texttt{docker run} for VEP and bcftools,
|
|
||||||
and a container platform gives you a container, not a Docker daemon. Rather than leave a button
|
|
||||||
that always fails, the cases are annotated where the pipeline works and copied up, and the
|
|
||||||
pipeline actions are hidden. Making it idle correctly needed one real change --- the platform
|
|
||||||
decides a service is idle from its \emph{outbound} traffic, and a pooled database connection is
|
|
||||||
outbound traffic, so the default connection pool would have kept the service awake and billable
|
|
||||||
for ever.
|
|
||||||
|
|
||||||
\section{What is still wrong}
|
|
||||||
|
|
||||||
An honest introduction should end with the open problems, not the achievements.
|
|
||||||
|
|
||||||
\begin{itemize}
|
|
||||||
\item \textbf{The model has no features in its training data.} Its only two remaining inputs,
|
|
||||||
CADD and AlphaMissense, are absent from all 688,362 training rows. It abstains today, so
|
|
||||||
nothing is broken --- but installing the VEP plugins would flip it on while it still knows
|
|
||||||
nothing. Serving should refuse unless the model was trained on the features it is handed.
|
|
||||||
\item \textbf{The benchmark is contaminated} and no amount of code fixes it. A leave-one-publication-out
|
|
||||||
rebuild is possible --- HPO's annotation file carries the source PMID ---
|
|
||||||
and is the honest next step.
|
|
||||||
\item \textbf{No baseline comparison.} Scoring the same held-out rows with CADD and AlphaMissense
|
|
||||||
would give the model something to beat. AlphaMissense is a 0.64\,GB download and tractable;
|
|
||||||
CADD's whole-genome file is 87.5\,GB and is not.
|
|
||||||
\item \textbf{Rarity contributes nothing without the cache} in the default demonstration, though
|
|
||||||
the gnomAD streaming route above now makes that solvable.
|
|
||||||
\end{itemize}
|
|
||||||
|
|
||||||
\vspace{1.4em}
|
|
||||||
\hrule
|
|
||||||
\vspace{0.9em}
|
|
||||||
{\small\color{soft}
|
|
||||||
rarelens is a self-training project built in the open on public data, licensed AGPL-3.0. It is not
|
|
||||||
a clinical tool and makes no diagnostic claim: ACMG/AMP treats computational predictions as
|
|
||||||
supporting evidence only, never sufficient alone. Data sources, licences, citations and the
|
|
||||||
evaluation caveats are documented in \texttt{docs/data.md}; the architecture decisions, including
|
|
||||||
why Google Cloud rather than AWS, are in \texttt{docs/architecture.md} and \texttt{docs/cloud.md}.}
|
|
||||||
|
|
||||||
\end{document}
|
|
||||||