Table of Contents
Execution backends
POST /cases/{id}/annotate writes a jobs row and hands off. app/services/events.launch()
then picks a backend from configuration alone — the pipeline code is identical in all three cases.
| Setting present | Backend | Behaviour |
|---|---|---|
cloudrun_job |
Cloud Run job | executes the Nextflow driver with argument overrides via the Jobs API |
pubsub_topic |
Pub/Sub | publishes the job; Argo Events triggers an Argo Workflow |
| neither | local subprocess | runs Nextflow directly, streaming stdout into the job log |
They are checked in that order.
Why three
Cloud Run job is the default because it scales to zero: nothing runs between pipelines. The
service account holds run.jobsExecutorWithOverrides on exactly one job, not project-wide.
Pub/Sub is the decoupled path. Retries, ordering and back-pressure become the queue's problem instead of the API's, and the API can restart mid-pipeline without losing work. This is the right shape once more than one thing produces annotation requests.
The local subprocess is what a developer gets with nothing configured. _watch streams the
last PROGRESS_LINES of Nextflow's output into the job row every PROGRESS_INTERVAL_S seconds so
the UI can show live progress, and _record_progress only writes while the job is still running,
so a late write cannot resurrect a finished job.
All three converge on the same jobs row. The interface polls one endpoint and does not know or
care which ran.
When the API has no Nextflow
If shutil.which("nextflow") finds nothing, the job is marked failed with the command to run by
hand. That is the normal state in docker-compose, and on Railway — see Deployments.
rarelens
Understanding it
Working on it
Running it
When it goes wrong