kemal/rarelens
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: a technical introduction to the platform, as a LaTeX article

Browse Source 
docs/blog/rarelens.tex builds a 10-page write-up covering the Nextflow DSL2
pipeline, the three execution backends behind one API call, the Argo and
ArgoCD track, Terraform, the external systems integrated, the ranking, the
benchmark and the model.

It is deliberately as much an account of what was wrong as of what works: the
components that scored evidence nobody had looked up, the missense AUROC that
fell from 0.872 to 0.500 once allele frequency was removed, the ontology walk
that silently dropped 399 terms, and the propagation change that measured
slightly worse than what it replaced and was kept anyway with the numbers
published.

Figures come from one headless-browser script. Screenshots are of the running
application; the six diagrams are hand-written HTML styled from the same
palette, rendered by the same script. Keeping both in one place is what stops
the article drifting from the system, and every number in a diagram has to be
changed deliberately.
This commit is contained in:
Kemal Yaylali
2026-09-12 13:10:47 +01:00
parent c25fb53666
commit 1e46fff2ff
25 changed files with 729 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/blog/diagram-sources/pipeline.html
+18
View File
@@ -0,0 +1,18 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<div id="frame">
<h1>The Nextflow DSL2 pipeline</h1>
<p class="sub">Three processes, each a container. The same workflow file runs on a laptop, on Google Batch and under Argo Workflows &mdash; the executor is a profile, not a rewrite.</p>
<div class="row" style="align-items:stretch">
<div class="box" style="flex:1"><div class="t">NORMALISE</div><div class="d">bcftools norm: left-align and split multi-allelics, then set the VCF ID to <span class="k" style="font-family:var(--mono)">CHROM_POS_REF_ALT</span>.</div><div class="k">bcftools 1.20</div></div>
<div class="arrow">&#8594;</div>
<div class="box amber" style="flex:1.25"><div class="t">VEP</div><div class="d">Ensembl VEP 113. Cache mode for full annotation, or <span class="k" style="font-family:var(--mono)">--database</span> against Ensembl's public server when the 25&nbsp;GB cache is not available. CADD and AlphaMissense are optional plugins.</div><div class="k">4 cpus, 8 GB</div></div>
<div class="arrow">&#8594;</div>
<div class="box" style="flex:1"><div class="t">LOAD_DB</div><div class="d">Parses VEP <span class="k" style="font-family:var(--mono)">--tab</span> output, inserts variants idempotently (a retried task cannot duplicate), and marks the job succeeded.</div><div class="k">psycopg, chunked COPY</div></div>
</div>
<div style="height:18px"></div>
<div class="row">
<div class="box plum" style="flex:1"><div class="t">Why the ID carries identity</div><div class="d">VEP's own Location and Allele columns trim indel alleles and shift positions, so a deletion round-trips as the wrong variant. Carrying <span class="k" style="font-family:var(--mono)">CHROM_POS_REF_ALT</span> through the ID field keeps the exact alleles the caller emitted. Verified on a real run: <span class="k" style="font-family:var(--mono)">22:42126611 CT&gt;C</span> survives intact.</div></div>
<div style="width:14px"></div>
<div class="box green" style="flex:1"><div class="t">Stub blocks keep CI honest</div><div class="d">Every process has a <span class="k" style="font-family:var(--mono)">stub:</span> block, so CI runs <span class="k" style="font-family:var(--mono)">nextflow run main.nf -stub-run</span> on a three-record fixture and checks wiring and channel shapes on every pull request &mdash; no containers, no VEP cache, seconds not hours.</div></div>
</div>
</div>