fix: overhaul the platform skeleton, add a serverless deployment track

An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.

CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)

ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
  so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
  version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms

Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch

Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
  Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
  for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
  Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.

Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
  VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes

Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
  gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice

Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.
This commit is contained in:
Kemal Yaylali
2026-09-12 07:21:11 +01:00
parent 5463f489a3
commit 11fb6b3d73
100 changed files with 3431 additions and 340 deletions
+38 -3
View File
@@ -1,8 +1,12 @@
from datetime import datetime
import re
import uuid
from datetime import datetime
from pathlib import PurePosixPath
from typing import Literal
from pydantic import BaseModel, ConfigDict, Field
from pydantic import BaseModel, ConfigDict, Field, field_validator
from app.config import settings
from app.models import JobStatus
@@ -10,10 +14,34 @@ class ORMModel(BaseModel):
model_config = ConfigDict(from_attributes=True)
Assembly = Literal["GRCh38", "GRCh37"]
VCF_SUFFIXES = (".vcf", ".vcf.gz", ".vcf.bgz", ".bcf")
GCS_URI = re.compile(r"gs://[a-z0-9][a-z0-9._-]{1,220}[a-z0-9]/\S+")
class SampleCreate(BaseModel):
name: str = Field(min_length=1, max_length=120)
vcf_uri: str
assembly: str = "GRCh38"
# Passed to VEP --assembly; the VEP cache must contain it.
assembly: Assembly = "GRCh38"
@field_validator("vcf_uri")
@classmethod
def _gcs_object_or_file_under_data_root(cls, v: str) -> str:
# The URI becomes a Nextflow argument and a path the pipeline reads: accept a GCS object or
# a file under the local data root, never something that parses as an option.
if not v or any(ord(c) < 32 for c in v):
raise ValueError("vcf_uri must be a non-empty single line")
if not v.lower().endswith(VCF_SUFFIXES):
raise ValueError(f"vcf_uri must end in one of {', '.join(VCF_SUFFIXES)}")
if v.startswith("gs://"):
if not GCS_URI.fullmatch(v):
raise ValueError("vcf_uri is not a valid gs://bucket/object URI")
return v
path, root = PurePosixPath(v), PurePosixPath(settings.local_data_root)
if not path.is_absolute() or ".." in path.parts or not path.is_relative_to(root):
raise ValueError(f"local VCFs must be absolute paths under {root}")
return v
class SampleOut(ORMModel):
@@ -30,6 +58,7 @@ class JobOut(ORMModel):
status: JobStatus
workflow_ref: str | None
vep_version: str | None
log: str | None
created_at: datetime
finished_at: datetime | None
@@ -56,6 +85,12 @@ class VariantOut(ORMModel):
prediction: PredictionOut | None = None
class ScoreOut(BaseModel):
job_id: uuid.UUID
scored: int
model_version: str
class VariantPage(BaseModel):
items: list[VariantOut]
total: int