Gotchas

Things that cost real debugging time here. Each one is a bug that was found and fixed; they are recorded because the symptom rarely points at the cause.

Science

A missing measurement is not a measurement of zero. rarity_score(None) read "this run has no allele frequencies" as "absent from gnomAD, therefore maximally rare" and handed every variant a free 0.25. The fix is structural: jobs record what they looked up, and a component with no evidence abstains. If you add a component, decide what makes it abstain first. See Ranking.

A DAG walked pre-order and read backwards loses ancestors. Computing each HPO term's ancestors with a pre-order DFS reversed looks like a post-order, but on a DAG a term can be reached before one of its parents, and then inherits that parent alone instead of the parent's whole lineage. It silently dropped 399 terms out of the phenotype branch — including Camptodactyly and Chiari malformation — so cases mentioning them scored lower for no visible reason. Now a true post-order, tested against a reference transitive closure.

A feature can cause its own label. The model took allele frequency as a feature, while ACMG's BA1/BS1 criteria assign ClinVar's benign labels using allele frequency. It was rediscovering the rule that generated its training data, and the impressive missense AUROC of 0.872 fell to exactly 0.500 when the feature was removed. See Benchmarks.

Split by gene, not by variant. A random split puts variants of the same gene on both sides and the model learns the gene. Grimm et al. documented this inflation for exactly this class of tool.

Pipeline

VEP's coordinates are not a primary key. Location and Allele trim indel alleles and shift positions; CT>C comes back as -> elsewhere. Identity travels in the VCF ID instead. See Pipeline.

--af_gnomade is rejected with --database, and plain --af silently returns nothing even for a variant at 15% global frequency. Frequencies genuinely need the cache — or a range request against gnomAD's public bucket, see Data sources.

Nextflow rejects two staged inputs sharing a filename. The cache placeholder and the plugin placeholder must be different files (assets/NO_CACHE and assets/NO_FILE), or the run fails with an input collision that names neither.

head plus pipefail kills a script silently. head closes the pipe, the upstream process takes SIGPIPE, and with pipefail the whole script aborts with no message.

Infrastructure

Kustomize only substitutes hashed ConfigMap names in workloads it considers in scope. Missing namespace: in an overlay means the hash is generated but never substituted, and pods mount a name that does not exist. The symptom is CreateContainerConfigError with correct-looking manifests.

A connection pool stops a container ever sleeping. Platforms that sleep idle containers measure outbound traffic; a pooled database connection is outbound traffic. See Deployments.

Local environment

macOS AirPlay Receiver owns port 5000 and answers 403, which looks exactly like an auth failure from MLflow. MLflow is on 5001 here.

*.rlwy.net may resolve to 0.0.0.0 on a machine with DNS filtering, so Railway's TCP proxy looks refused when it is fine. Check with a public resolver and connect by IP.

Security boundaries, not conveniences

vcf_uri validation is a boundary. A case's URI must be a gs:// object or an absolute path beneath LOCAL_DATA_ROOT, ending in a VCF suffix. Without it, a crafted value becomes a Nextflow option or reads an arbitrary file. It has its own tests; do not relax it for convenience.

The database URL never goes on a command line. Environment or Nextflow secret only, so it stays out of .command.sh and the workflow logs.

rarelens

Understanding it

Working on it

Running it

When it goes wrong