Three services -- Postgres, API, UI -- with the API on Railway's private
network only, so the UI's /api proxy is the single public entry point and
there is no CORS.
The pipeline cannot run there. Nextflow shells out to `docker run` for VEP
and bcftools, and Railway gives you a container, not a Docker daemon. Rather
than leave a button that always fails, cases are annotated locally and copied
up by scripts/seed-remote.sh, and PUBLIC_PIPELINE_ENABLED=false hides the
analyse/score actions and the create-case form.
DATABASE_IDLE_CONNECTIONS=false is what makes idling work. Railway decides a
service is idle from its *outbound* traffic and sleeps it after ~5-10 minutes;
a pooled database connection is outbound traffic, so SQLAlchemy's default pool
would have kept the API awake and billable for ever. Setting it false switches
to NullPool, which costs a connection per request -- nothing at demo traffic,
the wrong trade under real load, hence the flag rather than a rewrite.
BASIC_AUTH_USER / BASIC_AUTH_PASSWORD put one shared credential in front of
the site. Nothing deployed is patient data, so this stops the URL being
wandered into rather than protecting anyone's privacy; unset, the site is
open, which is what local development wants. Compared in constant time, and
both halves of the credential are checked even when the first fails.
From clicking through the redesigned UI:
- scoring a case without a model registry painted a red failure across a case that had in
fact analysed fine. It is now a quiet note saying the model term contributes 0, because
scoring is an optional fourth of the rank, not the analysis.
- the MLflow default moves to port 5001. On macOS, AirPlay Receiver owns 5000, which is why
the registry answered "403" rather than refusing the connection; docker-compose publishes
5001 to match.
- a funnel step that kept nothing drew a visible bar. Zero now draws zero.
- "1 candidates".
- the funnel's fixed grid columns forced a horizontal scrollbar on the report.
The report also lists the top undecided candidates now: the first thing anyone opens has no
decisions in it, and "Shortlisted (0)" alone said nothing about what the tool found.
Tests: api 77, web 32; ruff, mypy, svelte-check clean.
Makes a real annotation runnable locally without the 25 GB VEP cache, which is what
the demo needs and what a reviewer can reproduce in minutes.
- params.vep_database (VEP_DATABASE=true) queries Ensembl's public database instead of
a local cache. Slower per variant and fewer fields, so --everything is swapped for the
flags the loader actually stores. Its cache placeholder is NO_CACHE, not NO_FILE:
Nextflow rejects two staged inputs sharing a filename.
- PIPELINE_DATABASE_URL is handed to the pipeline when set. The loader runs inside a
container, where the API's own localhost URL would point at the container itself.
- README: how to run the UI's annotate button locally against host Nextflow + Docker.
Verified end to end on pipeline/tests/data/tiny.vcf: bcftools norm split the multiallelic
record, VEP 113 annotated 4 variants live, the loader wrote them and marked the job
succeeded, and the UI shows them. The deletion came back as 22:42126611 CT>C with exact
VCF alleles, which is the case the audit's ID-tagging fix exists for.
Tests: api 51, loader 16, stub run 3/3; ruff, mypy clean.
An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.
CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)
ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms
Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch
Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.
Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes
Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice
Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.
Kemal Yaylali
kemal