Kemal Yaylali
11fb6b3d73
An end-to-end audit found the repo could not build, test or run as shipped. This fixes every finding, then adds a Cloud Run track so the demo costs about £1/month idle instead of ~£150. CI (red on its first run) - api: setuptools could not build the package (flat layout with app/ and alembic/) - web: missing @types/node; `vitest run` exited 1 with no test files - pipeline: the stub run needed a gitignored VCF, and no process had a stub block - ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI) ML serving (scores were meaningless) - the registered model now carries its own feature engineering and returns predict_proba, so serving sends raw columns and cannot drift from training - resolve by registry alias (stages are deprecated in MLflow 3) and record the real version; re-scoring upserts instead of failing on the unique constraint - ClinVar labels parsed from VEP's lowercase terms Pipeline - exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent - job status reaches running/failed/succeeded, so the UI stops polling dead jobs - DATABASE_URL travels in the environment or a Nextflow secret, never on a command line - VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch Deployment - the API serves /api (matching the ingress); the web app reads its API URL at runtime - migrations run in an init container under a Postgres advisory lock - terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload Identity bindings, Secret Manager, deletion protection - serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md. Correctness and security - 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider VEP text columns, enum dropped on downgrade, the sample's assembly actually used - vcf_uri restricted to gs:// objects or files under the data root, blocking option injection - CORS restricted to configured origins; `make down` no longer deletes volumes Data - docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar, gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform validate and both kustomize overlays clean.
5.8 KiB
Data: what rarelens actually runs on
Everything below is public, peer-reviewed and consented for open redistribution. No patient data, no data access agreement, nothing that needs an application. These are the references to quote when showing the platform to someone.
Citations were verified against PubMed; each row links its DOI.
The demo slice
make data fetches two real files, chromosome 22 only (roughly 100 MB, minutes rather than hours):
| File | What it is | Role |
|---|---|---|
data/example.vcf.gz |
GIAB HG002 (NA24385) v4.2.1 benchmark calls, GRCh38, chr22 | the sample a scientist annotates |
data/clinvar.chr22.vcf.gz |
ClinVar, GRCh38, chr22 | training labels, and the ClinVar column in the UI |
HG002 is the NIST Genome in a Bottle Ashkenazi son, recruited through the Personal Genome Project, which consents participants to unrestricted public release. It is the reference genome the field benchmarks variant callers against, so it is both realistic and unambiguously shareable.
Datasets
| Dataset | Used for | Access | Terms | Citation |
|---|---|---|---|---|
| ClinVar (GRCh38) | pathogenic/benign labels, ClinVar column | ftp.ncbi.nlm.nih.gov/pub/clinvar/vcf_GRCh38/ |
NCBI public domain | Landrum et al., Nucleic Acids Res 48(D1):D835–D844, 2020. 10.1093/nar/gkz972 |
| Genome in a Bottle HG002 v4.2.1 | the demo sample | ftp-trace.ncbi.nlm.nih.gov/ReferenceSamples/giab/, s3://giab |
open, no use restriction | Zook et al., Nat Biotechnol 37:561–566, 2019. 10.1038/s41587-019-0074-6 |
| gnomAD v4 | allele frequency feature and filter | gs://gcp-public-data--gnomad, s3://gnomad-public-us-east-1 |
free use, no restriction | Chen et al., Nature 625:92–100, 2024. 10.1038/s41586-023-06045-0; Karczewski et al., Nature 581:434–443, 2020. 10.1038/s41586-020-2308-7 |
| 1000 Genomes 30x | optional cohort/trio data | EBI FTP, s3://1000genomes |
fully open, no access restriction | Byrska-Bishop et al., Cell 185(18):3426–3440.e19, 2022. 10.1016/j.cell.2022.08.004 |
| MANE Select | one transcript per gene, if transcript choice ever matters | Ensembl/RefSeq | open | Morales et al., Nature 604:310–315, 2022. 10.1038/s41586-022-04558-8 |
Tools and scores
| Tool | Role | Terms | Citation |
|---|---|---|---|
| Ensembl VEP 113 | annotation (pipeline/modules/vep.nf) |
Apache 2.0 | McLaren et al., Genome Biol 17:122, 2016. 10.1186/s13059-016-0974-4 |
| CADD | cadd_phred feature |
free for non-commercial use; commercial licence required | Rentzsch et al., Nucleic Acids Res 47(D1):D886–D894, 2019. 10.1093/nar/gky1016; Schubach et al., Nucleic Acids Res 52(D1), 2024. 10.1093/nar/gkad989 |
| AlphaMissense | am_pathogenicity feature |
predictions moved to CC BY 4.0 in March 2024 (originally CC BY-NC-SA) | Cheng et al., Science 381:eadg7492, 2023. 10.1126/science.adg7492 |
Neither score is required: rarelens_ml.features treats a missing CADD or AlphaMissense value as
NaN and LightGBM handles it, so the pipeline runs without the plugin data.
Evaluating the model honestly
The model trains on ClinVar labels and is scored on ClinVar-labelled variants, which is exactly where published benchmarks go wrong. What to do about it:
- Never let the label into the features.
CLIN_SIGis excluded by construction;clinvar_sigis stored for display only (rarelens_ml/features.pylists the five feature columns). - Split by gene, not by variant. Random splits put variants from the same gene on both sides, and a model can then score a gene rather than a variant. Grimm et al. showed this inflates reported accuracy for exactly this class of tool: Hum Mutat 36:513–523, 2015. 10.1002/humu.22768
- Prefer a time-based holdout. Train on an older ClinVar release (monthly archives live under
vcf_GRCh38/archive_2.0/) and test only on variants classified after that date. This is the closest thing to a prospective evaluation available without new patients. - Filter labels by review status. ClinVar's
CLNREVSTATmarks how much evidence backs a classification; two-star and above ("multiple submitters, no conflicts") is the usual bar. Known gap: VEP'sCLIN_SIGdoes not carry review status, so this needs ClinVar annotated as a custom field before it can be enforced. - Report against published baselines on the same rows. CADD PHRED and AlphaMissense are already columns in the variant table, so AUROC and AUPRC for the model next to those two, with the variant count, is a fair comparison rather than a number with nothing to beat.
- Report AUPRC, not just AUROC. Pathogenic variants are the minority class; AUROC flatters.
What must not be claimed
ACMG/AMP treats computational predictions as supporting evidence only, never sufficient on their own for classifying a variant (Richards et al., Genet Med 17:405–424, 2015. 10.1038/gim.2015.30). rarelens is a learning platform on public data: it makes no diagnostic claim, and the UI shows a score next to the evidence rather than a verdict. For what a real diagnostic pipeline looks like end to end, see the 100,000 Genomes Project rare-disease pilot: Smedley et al., N Engl J Med 385:1868–1880, 2021. 10.1056/NEJMoa2035790