fix: overhaul the platform skeleton, add a serverless deployment track

An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.

CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)

ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
  so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
  version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms

Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch

Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
  Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
  for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
  Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.

Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
  VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes

Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
  gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice

Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.

ml/Dockerfile

@@ -2,6 +2,6 @@ FROM python:3.12-slim
 WORKDIR /ml
 RUN pip install --no-cache-dir uv
 COPY pyproject.toml .
-RUN uv pip install --system -e .
+RUN uv pip install --system -r pyproject.toml
 COPY rarelens_ml ./rarelens_ml
 ENTRYPOINT ["python", "-m", "rarelens_ml.train"]

ml/pyproject.toml

@@ -1,8 +1,21 @@
+[build-system]
+requires = ["setuptools>=69"]
+build-backend = "setuptools.build_meta"
+
 [project]
 name = "rarelens-ml"
 version = "0.1.0"
 requires-python = ">=3.12"
-dependencies = ["lightgbm>=4.5", "mlflow>=2.16", "pandas", "scikit-learn", "sqlalchemy", "psycopg[binary]"]
+# mlflow major must match the API's mlflow-skinny and the tracking server image.
+dependencies = ["lightgbm>=4.5", "mlflow>=3,<4", "pandas", "scikit-learn"]
 
 [project.optional-dependencies]
 gpu = ["torch"]   # for the optional deep-learning baseline on GPU
+dev = ["pytest>=8"]
+
+[tool.setuptools.packages.find]
+include = ["rarelens_ml*"]
+
+[tool.pytest.ini_options]
+pythonpath = ["."]
+testpaths = ["tests"]

ml/rarelens_ml/features.py

@@ -1,14 +1,20 @@
-"""Feature engineering shared by training and serving. Keep this identical to api/app/services/scoring.py."""
+"""Feature engineering: the only copy.
+
+Training imports it, and train.log_and_register ships this package inside the logged pyfunc
+(code_paths), so serving runs exactly this code on the raw columns below.
+"""
 import pandas as pd
 
+# What serving must send: raw values as stored in the variants table / its annotations.
+RAW_COLUMNS = ["impact", "consequence", "gnomad_af", "cadd_phred", "am_pathogenicity"]
+
 IMPACT_ORDER = {"MODIFIER": 0, "LOW": 1, "MODERATE": 2, "HIGH": 3}
-CATEGORICAL = ["consequence"]
-NUMERIC = ["impact_rank", "gnomad_af", "cadd_phred", "am_pathogenicity"]
 
 
 def build(df: pd.DataFrame) -> pd.DataFrame:
-    out = pd.DataFrame()
-    out["impact_rank"] = df["impact"].map(IMPACT_ORDER).fillna(0)
+    out = pd.DataFrame(index=df.index)
+    out["impact_rank"] = df["impact"].map(IMPACT_ORDER).fillna(0).astype(int)
+    # No gnomAD record means the variant was not observed: treat as AF 0.
     out["gnomad_af"] = pd.to_numeric(df["gnomad_af"], errors="coerce").fillna(0.0)
     out["cadd_phred"] = pd.to_numeric(df["cadd_phred"], errors="coerce")
     out["am_pathogenicity"] = pd.to_numeric(df["am_pathogenicity"], errors="coerce")

ml/rarelens_ml/model.py

@@ -0,0 +1,17 @@
+import mlflow
+
+from rarelens_ml.features import RAW_COLUMNS, build
+
+
+class PathogenicityModel(mlflow.pyfunc.PythonModel):
+    """Serving contract: raw VEP columns in, P(pathogenic) out.
+
+    The stock LightGBM pyfunc flavour calls `predict`, which returns class labels; wrapping the
+    classifier keeps feature engineering and `predict_proba` inside the registered artifact.
+    """
+
+    def __init__(self, classifier):
+        self.classifier = classifier
+
+    def predict(self, context, model_input, params=None):
+        return self.classifier.predict_proba(build(model_input[RAW_COLUMNS]))[:, 1]

ml/rarelens_ml/train.py

@@ -1,54 +1,130 @@
-"""Train a pathogenicity classifier on ClinVar labels (Pathogenic/Likely pathogenic vs Benign/Likely benign).
+"""Train a pathogenicity classifier on ClinVar labels ((likely) pathogenic vs (likely) benign).
 
 Label leakage warning: CLIN_SIG must never be a feature. This is a learning exercise, not a clinical model.
-Usage: python -m rarelens_ml.train --tsv results/clinvar.vep.tsv
+Usage: python -m rarelens_ml.train --tsv results/clinvar.vep.tsv --register
 """
 import argparse
+import re
+from pathlib import Path
 
 import lightgbm as lgb
 import mlflow
-import mlflow.lightgbm
 import pandas as pd
+import sklearn
+from mlflow import MlflowClient
 from sklearn.metrics import average_precision_score, roc_auc_score
 from sklearn.model_selection import train_test_split
 
-from rarelens_ml.features import build
+from rarelens_ml.features import RAW_COLUMNS, build
+from rarelens_ml.model import PathogenicityModel
 
-POS = {"Pathogenic", "Likely_pathogenic", "Pathogenic/Likely_pathogenic"}
-NEG = {"Benign", "Likely_benign", "Benign/Likely_benign"}
+PACKAGE_DIR = Path(__file__).resolve().parent
+MODEL_NAME = "rarelens-pathogenicity"
+PARAMS = {
+    "n_estimators": 400, "learning_rate": 0.05, "num_leaves": 31, "class_weight": "balanced",
+    "verbose": -1,
+}
+
+POS = {"pathogenic", "likely_pathogenic"}
+NEG = {"benign", "likely_benign"}
+# VEP --tab column -> raw feature column (am_pathogenicity already matches).
+VEP_TO_RAW = {
+    "IMPACT": "impact", "Consequence": "consequence", "gnomADe_AF": "gnomad_af",
+    "CADD_PHRED": "cadd_phred",
+}
+
+
+def label(clin_sig: object) -> int | None:
+    """1 / 0 when every ClinVar term agrees, None for VUS, conflicts and missing values.
+
+    Accepts VEP's lowercase comma-separated form ("pathogenic,likely_pathogenic") and ClinVar's
+    CLNSIG form ("Pathogenic/Likely_pathogenic").
+    """
+    if not isinstance(clin_sig, str):
+        return None
+    terms = {t for t in re.split(r"[,&/|]", clin_sig.strip().lower()) if t and t != "-"}
+    if terms and terms <= POS:
+        return 1
+    if terms and terms <= NEG:
+        return 0
+    return None
+
+
+def read_vep_tab(path: str | Path) -> pd.DataFrame:
+    """Read VEP --tab output as strings, keeping "-" (VEP's missing marker) verbatim.
+
+    Skips the "##" preamble by position instead of comment="#", which would also cut any value
+    containing "#".
+    """
+    with open(path) as fh:
+        for n, line in enumerate(fh):
+            if line.startswith("#Uploaded_variation"):
+                break
+        else:
+            raise ValueError(f"{path}: no #Uploaded_variation header; is this VEP --tab output?")
+    df = pd.read_csv(path, sep="\t", skiprows=n, dtype=str, keep_default_na=False)
+    return df.rename(columns={"#Uploaded_variation": "Uploaded_variation"})
 
 
 def load(tsv: str) -> tuple[pd.DataFrame, pd.Series]:
-    df = pd.read_csv(tsv, sep="\t", comment="#", header=None, dtype=str)
-    with open(tsv) as fh:
-        df.columns = next(l for l in fh if l.startswith("#Uploaded")).lstrip("#").rstrip().split("\t")
-    df = df.rename(columns={"IMPACT": "impact", "Consequence": "consequence", "gnomADe_AF": "gnomad_af",
-                            "CADD_PHRED": "cadd_phred"})
-    y = df["CLIN_SIG"].map(lambda s: 1 if s in POS else 0 if s in NEG else None)
+    df = read_vep_tab(tsv).rename(columns=VEP_TO_RAW)
+    for col in RAW_COLUMNS:  # plugin columns are absent when VEP ran without CADD/AlphaMissense
+        if col not in df:
+            df[col] = pd.NA
+    y = df["CLIN_SIG"].map(label)
     keep = y.notna()
-    return build(df[keep]), y[keep].astype(int)
+    return (
+        df.loc[keep, RAW_COLUMNS].reset_index(drop=True),
+        y[keep].astype(int).reset_index(drop=True),
+    )
+
+
+def fit(X: pd.DataFrame, y: pd.Series) -> lgb.LGBMClassifier:
+    return lgb.LGBMClassifier(**PARAMS).fit(build(X), y)
+
+
+def log_and_register(clf: lgb.LGBMClassifier, model_name: str, alias: str) -> str:
+    """Log the pyfunc, register it and point `alias` at the new version. Returns the version."""
+    info = mlflow.pyfunc.log_model(
+        name="model",
+        python_model=PathogenicityModel(clf),
+        code_paths=[str(PACKAGE_DIR)],
+        registered_model_name=model_name,
+        pip_requirements=[
+            f"lightgbm=={lgb.__version__}",
+            f"pandas=={pd.__version__}",
+            f"scikit-learn=={sklearn.__version__}",
+        ],
+    )
+    version = str(info.registered_model_version)
+    MlflowClient().set_registered_model_alias(model_name, alias, version)
+    return version
 
 
 def main() -> None:
     p = argparse.ArgumentParser()
     p.add_argument("--tsv", required=True)
-    p.add_argument("--register", action="store_true")
+    p.add_argument("--register", action="store_true",
+                   help="register the model and move the alias to the new version")
+    p.add_argument("--alias", default="production")
     a = p.parse_args()
 
     X, y = load(a.tsv)
     Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
 
-    mlflow.set_experiment("rarelens-pathogenicity")
+    mlflow.set_experiment(MODEL_NAME)
     with mlflow.start_run():
-        params = {"n_estimators": 400, "learning_rate": 0.05, "num_leaves": 31, "class_weight": "balanced"}
-        mlflow.log_params(params)
-        model = lgb.LGBMClassifier(**params).fit(Xtr, ytr)
-        proba = model.predict_proba(Xte)[:, 1]
-        mlflow.log_metrics({"auroc": roc_auc_score(yte, proba), "auprc": average_precision_score(yte, proba)})
-        mlflow.lightgbm.log_model(
-            model, "model",
-            registered_model_name="rarelens-pathogenicity" if a.register else None,
-        )
+        mlflow.log_params(PARAMS)
+        clf = fit(Xtr, ytr)
+        proba = clf.predict_proba(build(Xte))[:, 1]
+        mlflow.log_metrics({"auroc": roc_auc_score(yte, proba),
+                            "auprc": average_precision_score(yte, proba)})
+        if a.register:
+            version = log_and_register(clf, MODEL_NAME, a.alias)
+            print(f"registered {MODEL_NAME} v{version} as @{a.alias}")
+        else:
+            mlflow.pyfunc.log_model(name="model", python_model=PathogenicityModel(clf),
+                                    code_paths=[str(PACKAGE_DIR)])
 
 
 if __name__ == "__main__":

ml/tests/test_features.py

@@ -0,0 +1,38 @@
+import math
+
+import pandas as pd
+
+from rarelens_ml.features import RAW_COLUMNS, build
+
+
+def raw(**overrides: list) -> pd.DataFrame:
+    base = {
+        "impact": ["HIGH", "LOW", None],
+        "consequence": ["stop_gained", "synonymous_variant", None],
+        "gnomad_af": [None, "0.12", 0.001],
+        "cadd_phred": ["35", "2.1", "-"],
+        "am_pathogenicity": ["0.98", None, "-"],
+    }
+    base.update(overrides)
+    return pd.DataFrame(base, index=[10, 11, 12])
+
+
+def test_raw_columns_are_the_serving_contract() -> None:
+    assert RAW_COLUMNS == ["impact", "consequence", "gnomad_af", "cadd_phred", "am_pathogenicity"]
+
+
+def test_build_ranks_impact_and_coerces_numbers() -> None:
+    out = build(raw())
+    assert out["impact_rank"].tolist() == [3, 1, 0]
+    assert out["gnomad_af"].tolist() == [0.0, 0.12, 0.001]  # missing AF means absent from gnomAD
+    assert out["cadd_phred"].iloc[0] == 35.0
+    assert math.isnan(out["cadd_phred"].iloc[2])  # VEP writes "-" for missing
+    assert math.isnan(out["am_pathogenicity"].iloc[1])
+
+
+def test_build_keeps_the_input_index() -> None:
+    assert build(raw()).index.tolist() == [10, 11, 12]
+
+
+def test_build_makes_consequence_categorical() -> None:
+    assert isinstance(build(raw())["consequence"].dtype, pd.CategoricalDtype)

ml/tests/test_train.py

@@ -0,0 +1,104 @@
+from pathlib import Path
+
+import numpy as np
+import pandas as pd
+import pytest
+
+from rarelens_ml.train import label, read_vep_tab
+
+HEADER = [
+    "Uploaded_variation", "Location", "Allele", "Consequence", "IMPACT", "SYMBOL",
+    "gnomADe_AF", "CLIN_SIG", "CADD_PHRED", "am_pathogenicity",
+]
+
+
+def write_vep_tab(path: Path, rows: list[list[str]]) -> Path:
+    lines = [
+        "## ENSEMBL VARIANT EFFECT PREDICTOR v113.0",
+        "## Column descriptions:",
+        "#" + "\t".join(HEADER),
+        *("\t".join(r) for r in rows),
+    ]
+    path.write_text("\n".join(lines) + "\n")
+    return path
+
+
+@pytest.mark.parametrize(
+    ("clin_sig", "expected"),
+    [
+        # VEP writes lowercase, comma-separated terms from co-located ClinVar records.
+        ("pathogenic", 1),
+        ("pathogenic,likely_pathogenic", 1),
+        ("likely_benign", 0),
+        ("benign,likely_benign", 0),
+        # ClinVar VCF CLNSIG spelling must keep working too.
+        ("Pathogenic/Likely_pathogenic", 1),
+        ("Benign", 0),
+        ("uncertain_significance", None),
+        ("pathogenic,benign", None),  # conflicting evidence is not a label
+        ("-", None),
+        ("", None),
+        (np.nan, None),
+    ],
+)
+def test_label(clin_sig: object, expected: int | None) -> None:
+    assert label(clin_sig) == expected
+
+
+def test_read_vep_tab_uses_the_hash_header_and_keeps_dashes(tmp_path: Path) -> None:
+    tsv = write_vep_tab(
+        tmp_path / "x.vep.tsv",
+        [["22_1_A_G", "22:1", "G", "missense_variant", "MODERATE", "TBX1", "-", "pathogenic", "28", "0.9"]],
+    )
+    df = read_vep_tab(tsv)
+    assert list(df.columns) == HEADER
+    assert df.loc[0, "gnomADe_AF"] == "-"
+    assert df.loc[0, "CLIN_SIG"] == "pathogenic"
+
+
+def test_load_returns_raw_serving_columns_and_labels(tmp_path: Path) -> None:
+    from rarelens_ml.features import RAW_COLUMNS
+    from rarelens_ml.train import load
+
+    tsv = write_vep_tab(
+        tmp_path / "x.vep.tsv",
+        [
+            ["a", "22:1", "G", "missense_variant", "MODERATE", "TBX1", "0.0001", "pathogenic", "28", "0.9"],
+            ["b", "22:2", "A", "synonymous_variant", "LOW", "CHEK2", "0.12", "benign", "3", "-"],
+            ["c", "22:3", "T", "intron_variant", "MODIFIER", "CHEK2", "0.3", "uncertain_significance", "1", "-"],
+        ],
+    )
+    X, y = load(str(tsv))
+    assert list(X.columns) == RAW_COLUMNS
+    assert y.tolist() == [1, 0]  # the VUS row is dropped
+
+
+def test_logged_model_returns_probabilities_from_raw_columns(tmp_path: Path) -> None:
+    """The registered model must take the raw columns serving sends and return P(pathogenic)."""
+    import mlflow
+
+    from rarelens_ml.train import fit, log_and_register
+
+    rng = np.random.default_rng(0)
+    n = 400
+    impact = rng.choice(["HIGH", "MODERATE", "LOW", "MODIFIER"], n)
+    y = pd.Series(((impact == "HIGH") | (rng.random(n) < 0.1)).astype(int))
+    X = pd.DataFrame({
+        "impact": impact,
+        "consequence": rng.choice(["stop_gained", "missense_variant", "intron_variant"], n),
+        "gnomad_af": rng.random(n).round(4).astype(str),   # strings, as read from the DB
+        "cadd_phred": (rng.random(n) * 40).round(1).astype(str),
+        "am_pathogenicity": "-",
+    })
+
+    mlflow.set_tracking_uri(f"sqlite:///{tmp_path}/mlflow.db")
+    mlflow.set_experiment("test")
+    clf = fit(X, y)
+    version = log_and_register(clf, model_name="rarelens-test", alias="production")
+
+    model = mlflow.pyfunc.load_model("models:/rarelens-test@production")
+    scores = np.asarray(model.predict(X.head(50)))
+    assert version == "1"
+    assert scores.shape == (50,)
+    assert ((scores >= 0) & (scores <= 1)).all()
+    assert not set(np.unique(scores)) <= {0.0, 1.0}, "got class labels, expected probabilities"