Table of Contents
Pipeline
pipeline/main.nf is a three-process Nextflow DSL2 workflow. Each process runs in a pinned
container.
| Process | Container | Does |
|---|---|---|
NORMALISE |
bcftools 1.20 | left-align, split multi-allelics, set the VCF ID to CHROM_POS_REF_ALT |
VEP |
ensembl-vep 113.0 | consequence, gene, HGVS, ClinVar; optionally CADD and AlphaMissense |
LOAD_DB |
python + psycopg | parse VEP --tab, insert variants, mark the job succeeded |
Running it
# dry run: annotate without touching the database
make pipeline VCF=data/example.vcf.gz
# annotate for a real job created in the UI
make annotate JOB=<job id> VCF=data/example.vcf.gz
# no 25 GB VEP cache? query Ensembl's public database instead
VEP_DATABASE=true make pipeline VCF=pipeline/tests/data/tiny.vcf
Database mode takes roughly 25–35 seconds per variant and returns no allele frequencies and no plugin scores, so keep those runs to tens of variants. What that costs the ranking is described in Ranking.
Variant identity
VEP's Location and Allele columns cannot be used as a key. They trim indel alleles and
shift positions, so CT>C comes back as -> at a different coordinate. NORMALISE therefore
writes identity into the VCF ID, which VEP passes through untouched:
bcftools annotate --set-id '%CHROM\_%POS\_%REF\_%FIRST_ALT'
and load_db.parse_variant_id reads it back. It rsplits on _ from the right, because contig
names can contain underscores (chrUn_KI270742v1) while positions and alleles cannot.
Verified on a real run: 22:42126611 CT>C survives with its alleles intact.
Executors
The workflow never names an executor. pipeline/nextflow.config maps profiles:
- local / docker — a laptop, or the API's subprocess backend
- gcp — Google Batch, Spot VMs, needs
--projectand--bucket - Argo — the same containers as workflow steps, see Kubernetes and GitOps
Stub runs
Every process has a stub: block, so CI can run the whole workflow with no containers, no cache
and no database:
cd pipeline && nextflow run main.nf -stub-run --vcf tests/data/tiny.vcf
This runs on every pull request. It checks wiring and channel shapes — a renamed output, a process emitting the wrong cardinality — which is the class of breakage that otherwise only shows up in a two-hour annotation run.
Secrets
The database URL is passed by environment or as a Nextflow secret, never on a command line, so
it does not land in .command.sh or the workflow logs. PIPELINE_DATABASE_URL exists because the
loader runs inside a container where the API's own localhost would be the container itself.
rarelens
Understanding it
Working on it
Running it
When it goes wrong