fix: overhaul the platform skeleton, add a serverless deployment track

An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.

CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)

ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
  so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
  version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms

Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch

Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
  Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
  for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
  Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.

Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
  VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes

Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
  gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice

Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.

.github/workflows/ci.yml

@@ -19,7 +19,17 @@ jobs:
       - run: uv pip install --system -e "api[dev]"
       - run: cd api && ruff check . && mypy app
       - run: cd api && pytest -q
-        env: { DATABASE_URL: postgresql+asyncpg://rarelens:rarelens@localhost:5432/rarelens }
+        env:
+          DATABASE_URL: postgresql+asyncpg://rarelens:rarelens@localhost:5432/rarelens
+          REQUIRE_DB: "1"
+
+  ml:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v3
+      - run: uv pip install --system -e "ml[dev]"
+      - run: cd ml && pytest -q
 
   web:
     runs-on: ubuntu-latest
@@ -31,10 +41,23 @@ jobs:
 
   pipeline:
     runs-on: ubuntu-latest
+    services:
+      postgres:
+        image: postgres:16-alpine
+        env: { POSTGRES_USER: rarelens, POSTGRES_PASSWORD: rarelens, POSTGRES_DB: rarelens }
+        ports: ["5432:5432"]
+        options: --health-cmd "pg_isready -U rarelens" --health-interval 5s --health-retries 10
     steps:
       - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v3
+      - run: uv pip install --system -r pipeline/requirements.txt pytest
+      - run: cd pipeline && pytest -q tests
+        env:
+          DATABASE_URL: postgresql://rarelens:rarelens@localhost:5432/rarelens
+          REQUIRE_DB: "1"
       - uses: nf-core/setup-nextflow@v2
-      - run: cd pipeline && nextflow run main.nf -profile docker --vcf ../data/example.vcf.gz -stub-run
+      # Stub run checks wiring and channel shapes only; no containers or VEP cache needed.
+      - run: cd pipeline && nextflow run main.nf -stub-run --vcf tests/data/tiny.vcf
 
   terraform:
     runs-on: ubuntu-latest
@@ -45,11 +68,19 @@ jobs:
 
   images:
     if: github.ref == 'refs/heads/main'
-    needs: [api, web, pipeline]
+    needs: [api, ml, web, pipeline]
     runs-on: ubuntu-latest
     permissions: { contents: read, id-token: write }
     strategy:
-      matrix: { component: [api, web, pipeline, ml] }
+      matrix:
+        include:
+          - { component: api, context: api, dockerfile: api/Dockerfile }
+          - { component: web, context: web, dockerfile: web/Dockerfile }
+          - { component: ml, context: ml, dockerfile: ml/Dockerfile }
+          - { component: pipeline, context: pipeline, dockerfile: pipeline/Dockerfile }
+          - { component: loader, context: pipeline, dockerfile: pipeline/loader.Dockerfile }
+    env:
+      REGISTRY: europe-west2-docker.pkg.dev/${{ secrets.GCP_PROJECT }}/rarelens
     steps:
       - uses: actions/checkout@v4
       - uses: google-github-actions/auth@v2
@@ -59,6 +90,9 @@ jobs:
       - run: gcloud auth configure-docker europe-west2-docker.pkg.dev --quiet
       - uses: docker/build-push-action@v6
         with:
-          context: ${{ matrix.component }}
+          context: ${{ matrix.context }}
+          file: ${{ matrix.dockerfile }}
           push: true
-          tags: europe-west2-docker.pkg.dev/${{ secrets.GCP_PROJECT }}/rarelens/${{ matrix.component }}:${{ github.sha }}
+          # Only the pipeline driver uses it: pins the loader image from the same commit.
+          build-args: LOADER_IMAGE=${{ env.REGISTRY }}/loader:${{ github.sha }}
+          tags: ${{ env.REGISTRY }}/${{ matrix.component }}:${{ github.sha }}

.github/workflows/deploy.yml

@@ -10,12 +10,19 @@ jobs:
   bump:
     if: ${{ github.event.workflow_run.conclusion == 'success' }}
     runs-on: ubuntu-latest
+    permissions: { contents: write }
     steps:
       - uses: actions/checkout@v4
+      - name: Refuse to deploy an unconfigured overlay
+        run: |
+          if grep -rl __GCP_PROJECT__ infra/k8s/overlays/gcp infra/argo-workflows; then
+            echo "::error::run 'make gcp-configure PROJECT=<id>' and commit before deploying"
+            exit 1
+          fi
       - uses: imranismail/setup-kustomize@v2
       - run: |
           cd infra/k8s/overlays/gcp
-          for c in api web; do
+          for c in api web pipeline loader; do
             kustomize edit set image rarelens/$c=europe-west2-docker.pkg.dev/${{ secrets.GCP_PROJECT }}/rarelens/$c:${{ github.event.workflow_run.head_sha }}
           done
       - uses: stefanzweifel/git-auto-commit-action@v5