feat(ml): train a real model, and report the number that matters rather than the flattering one

"Variants are unscored" was accurate: nothing was ever trained, so a quarter of every rank was
dead weight and the UI leaked a connection error at the reader.

- scripts/make-training-set.sh derives a training table from ClinVar directly. ClinVar already
  carries the molecular consequence, the gene and an allele frequency, which is the feature set
  serving sends, so this avoids running VEP over hundreds of thousands of variants. 2-star
  records only.
- train.py now holds out whole genes (GroupShuffleSplit). docs/data.md had said to do this since
  the data pass; the code was still doing a random split, which is the leak Grimm 2015 describes.
- evaluate() reports missense on its own. On the last run: AUROC 0.986 over 74,239 held-out
  variants, but 0.872 over the 13,553 missense ones, and the docs say plainly why even that is
  flattered — within missense the only live feature is allele frequency, and ClinVar's benign
  calls often use allele frequency as evidence (ACMG BA1/BS1), so the feature partly caused the
  label.
- the 503 now names what is missing (model@alias via tracking URI) and leaves the exception in
  the server log instead of the UI.
- make training-set / make train; the 58 MB table is gitignored.

Verified end to end: model registered as v2, the simulated NF2 case scores 0.999 on the planted
variant, and it now ranks 1.00 with all four components live.

Tests: api 77, ml 22, loader 16, web 32; ruff, mypy, svelte-check clean.
This commit is contained in:
Kemal Yaylali
2026-09-12 09:13:54 +01:00
parent 3ab404ebe5
commit 197975cc42
9 changed files with 235 additions and 14 deletions
+8 -2
View File
@@ -8,6 +8,7 @@ from sqlalchemy import func, select
from sqlalchemy.exc import IntegrityError
from sqlalchemy.orm import selectinload
from app.config import settings
from app.db import SessionDep
from app.models import (
Case,
@@ -152,9 +153,14 @@ async def score(case_id: uuid.UUID, session: SessionDep):
scored, version = await score_job(job.id, session)
except Exception as e:
# No registry, no model behind the alias, a model that will not load: all of these are
# the environment being unready, not a bug in the request. Say so rather than throwing 500.
# the environment being unready, not a bug in the request. Name what is missing and keep
# the exception in the log, where it is useful, rather than in the UI, where it is noise.
logger.exception("scoring case %s failed", case_id)
raise HTTPException(503, f"could not score with the model: {e}") from e
raise HTTPException(
503,
f"no model available: {settings.model_name}@{settings.model_alias} "
f"via {settings.mlflow_tracking_uri}",
) from e
return ScoreOut(case_id=case_id, scored=scored, model_version=version)
+4 -1
View File
@@ -99,4 +99,7 @@ async def test_an_unreachable_model_registry_is_explained_not_a_500(
monkeypatch.setattr(scoring, "load_model", unreachable)
r = await client.post(f"/api/cases/{case_id}/score")
assert r.status_code == 503
assert "connection refused" in r.json()["detail"]
detail = r.json()["detail"]
# Names what is missing; the exception itself belongs in the server log, not the UI.
assert "rarelens-pathogenicity@production" in detail
assert "Max retries" not in detail and "Traceback" not in detail