"Variants are unscored" was accurate: nothing was ever trained, so a quarter of every rank was
dead weight and the UI leaked a connection error at the reader.
- scripts/make-training-set.sh derives a training table from ClinVar directly. ClinVar already
carries the molecular consequence, the gene and an allele frequency, which is the feature set
serving sends, so this avoids running VEP over hundreds of thousands of variants. 2-star
records only.
- train.py now holds out whole genes (GroupShuffleSplit). docs/data.md had said to do this since
the data pass; the code was still doing a random split, which is the leak Grimm 2015 describes.
- evaluate() reports missense on its own. On the last run: AUROC 0.986 over 74,239 held-out
variants, but 0.872 over the 13,553 missense ones, and the docs say plainly why even that is
flattered — within missense the only live feature is allele frequency, and ClinVar's benign
calls often use allele frequency as evidence (ACMG BA1/BS1), so the feature partly caused the
label.
- the 503 now names what is missing (model@alias via tracking URI) and leaves the exception in
the server log instead of the UI.
- make training-set / make train; the 58 MB table is gitignored.
Verified end to end: model registered as v2, the simulated NF2 case scores 0.999 on the planted
variant, and it now ranks 1.00 with all four components live.
Tests: api 77, ml 22, loader 16, web 32; ruff, mypy, svelte-check clean.
An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.
CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)
ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms
Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch
Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.
Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes
Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice
Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.
Kemal Yaylali