Kemal Yaylali
1e46fff2ff
docs/blog/rarelens.tex builds a 10-page write-up covering the Nextflow DSL2 pipeline, the three execution backends behind one API call, the Argo and ArgoCD track, Terraform, the external systems integrated, the ranking, the benchmark and the model. It is deliberately as much an account of what was wrong as of what works: the components that scored evidence nobody had looked up, the missense AUROC that fell from 0.872 to 0.500 once allele frequency was removed, the ontology walk that silently dropped 399 terms, and the propagation change that measured slightly worse than what it replaced and was kept anyway with the numbers published. Figures come from one headless-browser script. Screenshots are of the running application; the six diagrams are hand-written HTML styled from the same palette, rendered by the same script. Keeping both in one place is what stops the article drifting from the system, and every number in a diagram has to be changed deliberately.
23 lines
2.5 KiB
HTML
23 lines
2.5 KiB
HTML
<!doctype html><meta charset="utf-8"><link rel="stylesheet" href="_base.css">
|
|
<div id="frame">
|
|
<h1>One API call, three execution backends</h1>
|
|
<p class="sub"><span class="k" style="font-family:var(--mono)">POST /cases/{id}/annotate</span> creates a job row, then <span class="k" style="font-family:var(--mono)">events.launch()</span> picks a backend from configuration alone. The pipeline code never changes.</p>
|
|
|
|
<div class="row" style="align-items:flex-start">
|
|
<div class="box ink" style="flex:0 0 210px"><div class="t">events.launch(job)</div><div class="d">Chooses on settings, in order. Everything downstream writes back to the same <span class="k" style="font-family:var(--mono)">jobs</span> row, so the UI polls one endpoint whichever path ran.</div><div class="k">app/services/events.py</div></div>
|
|
<div class="arrow">→</div>
|
|
<div class="col" style="flex:1">
|
|
<div class="box amber"><div class="t">1. cloudrun_job set → Cloud Run job</div><div class="d">Executes the Nextflow driver as a job with argument overrides, via the Jobs API. Scales to zero between runs; the service account holds only <span class="k" style="font-family:var(--mono)">run.jobsExecutorWithOverrides</span> on that one job.</div></div>
|
|
<div class="box green"><div class="t">2. pubsub_topic set → Pub/Sub publish</div><div class="d">Publishes the job as an event. Argo Events' sensor subscribes and triggers an Argo Workflow on GKE. Decouples the API from the compute entirely; retries and back-pressure belong to the queue.</div></div>
|
|
<div class="box plum"><div class="t">3. otherwise → local subprocess</div><div class="d">Runs Nextflow directly and streams stdout into the job log, so the UI can show live progress. This is what a developer gets with no cloud configured at all.</div></div>
|
|
</div>
|
|
<div class="arrow">→</div>
|
|
<div class="col" style="flex:0 0 215px">
|
|
<div class="box"><div class="t">Nextflow pipeline</div><div class="d">NORMALISE → VEP → LOAD_DB</div></div>
|
|
<div class="arrow down">↓</div>
|
|
<div class="box"><div class="t">loader writes results</div><div class="d">Variants inserted, job marked succeeded, and the run records <em>what evidence it looked up</em>.</div><div class="k">has_frequencies, has_effect_scores</div></div>
|
|
</div>
|
|
</div>
|
|
<p class="note">The database URL never appears on a command line: it is passed by environment, or as a Nextflow secret, so it stays out of <span class="k" style="font-family:var(--mono)">.command.sh</span> and the workflow logs.</p>
|
|
</div>
|