Files
Kemal Yaylali 11fb6b3d73 fix: overhaul the platform skeleton, add a serverless deployment track
An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.

CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)

ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
  so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
  version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms

Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch

Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
  Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
  for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
  Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.

Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
  VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes

Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
  gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice

Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.
2026-09-12 07:21:11 +01:00

99 lines
3.6 KiB
YAML

name: ci
on:
pull_request:
push:
branches: [main]
jobs:
api:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env: { POSTGRES_USER: rarelens, POSTGRES_PASSWORD: rarelens, POSTGRES_DB: rarelens }
ports: ["5432:5432"]
options: --health-cmd "pg_isready -U rarelens" --health-interval 5s --health-retries 10
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v3
- run: uv pip install --system -e "api[dev]"
- run: cd api && ruff check . && mypy app
- run: cd api && pytest -q
env:
DATABASE_URL: postgresql+asyncpg://rarelens:rarelens@localhost:5432/rarelens
REQUIRE_DB: "1"
ml:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v3
- run: uv pip install --system -e "ml[dev]"
- run: cd ml && pytest -q
web:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: 22, cache: npm, cache-dependency-path: web/package-lock.json }
- run: cd web && npm ci && npm run check && npm test && npm run build
pipeline:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:16-alpine
env: { POSTGRES_USER: rarelens, POSTGRES_PASSWORD: rarelens, POSTGRES_DB: rarelens }
ports: ["5432:5432"]
options: --health-cmd "pg_isready -U rarelens" --health-interval 5s --health-retries 10
steps:
- uses: actions/checkout@v4
- uses: astral-sh/setup-uv@v3
- run: uv pip install --system -r pipeline/requirements.txt pytest
- run: cd pipeline && pytest -q tests
env:
DATABASE_URL: postgresql://rarelens:rarelens@localhost:5432/rarelens
REQUIRE_DB: "1"
- uses: nf-core/setup-nextflow@v2
# Stub run checks wiring and channel shapes only; no containers or VEP cache needed.
- run: cd pipeline && nextflow run main.nf -stub-run --vcf tests/data/tiny.vcf
terraform:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- run: cd infra/terraform && terraform init -backend=false && terraform fmt -check && terraform validate
images:
if: github.ref == 'refs/heads/main'
needs: [api, ml, web, pipeline]
runs-on: ubuntu-latest
permissions: { contents: read, id-token: write }
strategy:
matrix:
include:
- { component: api, context: api, dockerfile: api/Dockerfile }
- { component: web, context: web, dockerfile: web/Dockerfile }
- { component: ml, context: ml, dockerfile: ml/Dockerfile }
- { component: pipeline, context: pipeline, dockerfile: pipeline/Dockerfile }
- { component: loader, context: pipeline, dockerfile: pipeline/loader.Dockerfile }
env:
REGISTRY: europe-west2-docker.pkg.dev/${{ secrets.GCP_PROJECT }}/rarelens
steps:
- uses: actions/checkout@v4
- uses: google-github-actions/auth@v2
with:
workload_identity_provider: ${{ secrets.GCP_WIF_PROVIDER }}
service_account: ${{ secrets.GCP_CI_SA }}
- run: gcloud auth configure-docker europe-west2-docker.pkg.dev --quiet
- uses: docker/build-push-action@v6
with:
context: ${{ matrix.context }}
file: ${{ matrix.dockerfile }}
push: true
# Only the pipeline driver uses it: pins the loader image from the same commit.
build-args: LOADER_IMAGE=${{ env.REGISTRY }}/loader:${{ github.sha }}
tags: ${{ env.REGISTRY }}/${{ matrix.component }}:${{ github.sha }}