kemal/rarelens
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: a technical introduction to the platform, as a LaTeX article

Browse Source 
docs/blog/rarelens.tex builds a 10-page write-up covering the Nextflow DSL2
pipeline, the three execution backends behind one API call, the Argo and
ArgoCD track, Terraform, the external systems integrated, the ranking, the
benchmark and the model.

It is deliberately as much an account of what was wrong as of what works: the
components that scored evidence nobody had looked up, the missense AUROC that
fell from 0.872 to 0.500 once allele frequency was removed, the ontology walk
that silently dropped 399 terms, and the propagation change that measured
slightly worse than what it replaced and was kept anyway with the numbers
published.

Figures come from one headless-browser script. Screenshots are of the running
application; the six diagrams are hand-written HTML styled from the same
palette, rendered by the same script. Keeping both in one place is what stops
the article drifting from the system, and every number in a diagram has to be
changed deliberately.
This commit is contained in:
Kemal Yaylali
2026-09-12 13:10:47 +01:00
parent c25fb53666
commit 1e46fff2ff
25 changed files with 729 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/blog/diagram-sources/events.html
+22
View File
@@ -0,0 +1,22 @@
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
<div id="frame">
<h1>One API call, three execution backends</h1>
<p class="sub"><span class="k" style="font-family:var(--mono)">POST /cases/{id}/annotate</span> creates a job row, then <span class="k" style="font-family:var(--mono)">events.launch()</span> picks a backend from configuration alone. The pipeline code never changes.</p>
<div class="row" style="align-items:flex-start">
<div class="box ink" style="flex:0 0 210px"><div class="t">events.launch(job)</div><div class="d">Chooses on settings, in order. Everything downstream writes back to the same <span class="k" style="font-family:var(--mono)">jobs</span> row, so the UI polls one endpoint whichever path ran.</div><div class="k">app/services/events.py</div></div>
<div class="arrow">&#8594;</div>
<div class="col" style="flex:1">
<div class="box amber"><div class="t">1. cloudrun_job set &#8594; Cloud Run job</div><div class="d">Executes the Nextflow driver as a job with argument overrides, via the Jobs API. Scales to zero between runs; the service account holds only <span class="k" style="font-family:var(--mono)">run.jobsExecutorWithOverrides</span> on that one job.</div></div>
<div class="box green"><div class="t">2. pubsub_topic set &#8594; Pub/Sub publish</div><div class="d">Publishes the job as an event. Argo Events' sensor subscribes and triggers an Argo Workflow on GKE. Decouples the API from the compute entirely; retries and back-pressure belong to the queue.</div></div>
<div class="box plum"><div class="t">3. otherwise &#8594; local subprocess</div><div class="d">Runs Nextflow directly and streams stdout into the job log, so the UI can show live progress. This is what a developer gets with no cloud configured at all.</div></div>
</div>
<div class="arrow">&#8594;</div>
<div class="col" style="flex:0 0 215px">
<div class="box"><div class="t">Nextflow pipeline</div><div class="d">NORMALISE &#8594; VEP &#8594; LOAD_DB</div></div>
<div class="arrow down">&#8595;</div>
<div class="box"><div class="t">loader writes results</div><div class="d">Variants inserted, job marked succeeded, and the run records <em>what evidence it looked up</em>.</div><div class="k">has_frequencies, has_effect_scores</div></div>
</div>
</div>
<p class="note">The database URL never appears on a command line: it is passed by environment, or as a Nextflow secret, so it stays out of <span class="k" style="font-family:var(--mono)">.command.sh</span> and the workflow logs.</p>
</div>