fix(science): stop scoring evidence that was never looked up

A review of the ranking's arithmetic found four things wrong, all of which
made the score look better informed than it was. Measurements below are from
this repo, not estimates.

**Components now abstain instead of inventing a number.** A run without a VEP
cache returns no allele frequencies, and rarity_score(None) read that as
"absent from gnomAD, therefore maximally rare" and awarded every variant a
free 0.25. jobs.has_frequencies / has_effect_scores record what the run
actually produced, absent components are dropped from the weighted mean, and
the remaining weights are renormalised so the score keeps its meaning. The UI
shows "not looked up" rather than a bar, and the funnel stops calling a step
"rare" when nothing was filtered.

**Allele frequency is no longer a model feature.** It dominated: the same
missense variant scored 0.887 at AF 0 and 0.0003 at AF 0.01. That double-
counted, because the ranking already scores frequency explicitly, putting
~45% of every rank on one measurement; and it was circular, because ACMG
assigns ClinVar's benign labels using frequency (BA1/BS1). Retraining without
it moves missense AUROC from 0.872 to 0.500 — exactly random. The old figure
was allele frequency, not variant-effect knowledge. The model therefore
abstains unless CADD or AlphaMissense is present, since otherwise it only
restates the consequence class.

**Phenotype matching is weighted by information content** and HPO annotations
are propagated up the ontology. Counting terms alike let "global
developmental delay" (IC 0.93) count as much as "dilated left subclavian
artery" (IC 7.88).

**A real bug in the propagation, found by checking it.** The ancestor walk
read a pre-order DFS backwards, which on a DAG lets a term resolve before one
of its parents and inherit that parent alone instead of its lineage. It
dropped 399 terms out of the phenotype branch, Camptodactyly and Chiari
malformation among them. Now a true post-order, tested against a reference
transitive closure.

The ontology arithmetic moved to rarelens_ml.hpo so it is covered by tests,
and rarelens_ml.benchmark measures the whole thing: across 10,178 published
cases the causal gene ranks first 45.9-81.0% of the time against 5,269 genes,
versus 0.02% for chance. docs/data.md reports that with its contamination
(HPO's annotations come from these same case reports), and includes the
measurement showing information-content weighting earns its place while
propagation does not - kept anyway, for a reason the docs argue rather than
assume.

api/app/models.py

@@ -8,6 +8,7 @@ import uuid
 from datetime import datetime
 
 from sqlalchemy import (
+    Boolean,
     DateTime,
     Enum,
     Float,
@@ -63,8 +64,25 @@ class CasePhenotype(Base):
     case: Mapped[Case] = relationship(back_populates="phenotypes")
 
 
+class HpoTerm(Base):
+    """One HPO term and its information content (scripts/load-hpo.py); read-only reference data.
+
+    ic is -ln(fraction of annotated genes carrying the term), so a term shared by nearly every
+    gene is worth almost nothing and a near-pathognomonic one is worth a lot.
+    """
+
+    __tablename__ = "hpo_terms"
+    hpo_id: Mapped[str] = mapped_column(String(20), primary_key=True)
+    name: Mapped[str] = mapped_column(String(200))
+    ic: Mapped[float] = mapped_column(Float)
+
+
 class GenePhenotype(Base):
-    """HPO's gene-to-phenotype annotations (scripts/load-hpo.py); read-only reference data."""
+    """HPO's gene-to-phenotype annotations (scripts/load-hpo.py); read-only reference data.
+
+    Propagated up the ontology at load time: a gene annotated with a term also carries that term's
+    ancestors, so a case term matches a gene annotated with anything more specific.
+    """
 
     __tablename__ = "gene_phenotypes"
     __table_args__ = (UniqueConstraint("gene_symbol", "hpo_id", name="uq_gene_phenotypes_gene_term"),)
@@ -81,6 +99,10 @@ class Job(Base):
     status: Mapped[JobStatus] = mapped_column(Enum(JobStatus), default=JobStatus.queued)
     workflow_ref: Mapped[str | None] = mapped_column(String(200))   # Argo workflow name / nf run id
     vep_version: Mapped[str | None] = mapped_column(String(40))
+    # What this run's annotation actually produced. The ranking refuses to score a line of
+    # evidence the run never looked up, so it has to be recorded rather than assumed.
+    has_frequencies: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
+    has_effect_scores: Mapped[bool] = mapped_column(Boolean, default=False, server_default="false")
     log: Mapped[str | None] = mapped_column(Text)
     created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
     finished_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True))

api/app/routers/cases.py

@@ -24,6 +24,7 @@ from app.schemas import (
     CandidatePage,
     CaseCreate,
     CaseOut,
+    EvidenceOut,
     FunnelOut,
     JobOut,
     ProvenanceOut,
@@ -194,7 +195,8 @@ async def list_candidates(
     return CandidatePage(
         # The funnel describes the whole case, not the filtered view.
         funnel=FunnelOut(**asdict(view.funnel)),
-        weights=triage.WEIGHTS,
+        evidence=EvidenceOut(**asdict(view.evidence), missing=view.evidence.missing),
+        weights=triage.weights_in_use(view.evidence),
         items=[CandidateOut.from_candidate(c, labels) for c in items[offset : offset + limit]],
         total=len(items),
         limit=limit,

api/app/routers/variants.py

@@ -28,10 +28,15 @@ async def get_variant(variant_id: int, session: SessionDep) -> VariantDetailOut:
         raise HTTPException(404, "variant not found")
     case = await case_view.get_case(session, variant.job.case_id)
     terms = [p.hpo_id for p in case.phenotypes] if case else []
-    gene_terms = await case_view.gene_terms_for(session, {variant.gene} if variant.gene else set())
+    ontology = await case_view.ontology_for(
+        session, {variant.gene} if variant.gene else set(), terms
+    )
+    evidence = triage.Evidence(
+        frequencies=variant.job.has_frequencies, effect_scores=variant.job.has_effect_scores
+    )
 
     # evaluate, not rank: the panel must work for a variant that did not make the candidate list.
-    scored = triage.evaluate(variant, terms, gene_terms)
+    scored = triage.evaluate(variant, terms, ontology, evidence)
     labels = {p.hpo_id: p.label for p in case.phenotypes} if case else {}
     base = CandidateOut.from_candidate(scored, labels).model_dump()
     return VariantDetailOut(**base, annotations=variant.annotations or {})

api/app/schemas.py

@@ -114,7 +114,9 @@ class DecisionOut(ORMModel):
 class CandidateOut(BaseModel):
     variant: VariantOut
     score: float
-    components: dict[str, float]
+    # A null component means that evidence was never looked up, so it did not enter the score.
+    # It is not a zero, and the UI must not draw it as an empty bar.
+    components: dict[str, float | None]
     matched_terms: list[PhenotypeTerm]
     scored: bool
     decision: DecisionOut | None = None
@@ -125,7 +127,10 @@ class CandidateOut(BaseModel):
         return cls(
             variant=VariantOut.model_validate(variant),
             score=round(candidate.score, 4),
-            components={name: round(v, 4) for name, v in candidate.components.items()},
+            components={
+                name: None if v is None else round(v, 4)
+                for name, v in candidate.components.items()
+            },
             matched_terms=[
                 PhenotypeTerm(hpo_id=term, label=labels.get(term, term))
                 for term in candidate.matched_terms
@@ -144,10 +149,21 @@ class FunnelOut(BaseModel):
     rare: int
     candidates: int
     phenotype_matched: int
+    frequencies: bool = False  # False: the rare step could not filter, nothing was looked up
+
+
+class EvidenceOut(BaseModel):
+    """What the annotation run produced, and therefore which components scored at all."""
+
+    frequencies: bool
+    effect_scores: bool
+    missing: list[str]
 
 
 class CandidatePage(BaseModel):
     funnel: FunnelOut
+    evidence: EvidenceOut
+    # The weights as applied: renormalised over the components that had evidence.
     weights: dict[str, float]
     items: list[CandidateOut]
     total: int

api/app/services/candidates.py

@@ -4,13 +4,14 @@ Everything for one job is loaded at once, which is fine for a gene panel or a ch
 size of case this demo handles. A whole genome would need the narrowing pushed into SQL.
 """
 import uuid
-from dataclasses import dataclass
+from collections.abc import Sequence
+from dataclasses import dataclass, field
 
 from sqlalchemy import select
 from sqlalchemy.ext.asyncio import AsyncSession
 from sqlalchemy.orm import selectinload
 
-from app.models import Case, GenePhenotype, Job, JobStatus, Variant
+from app.models import Case, GenePhenotype, HpoTerm, Job, JobStatus, Variant
 from app.services import triage
 
 EMPTY_FUNNEL = triage.Funnel(total=0, rare=0, candidates=0, phenotype_matched=0)
@@ -22,6 +23,7 @@ class CaseView:
     job: Job | None
     funnel: triage.Funnel
     candidates: list[triage.Candidate]
+    evidence: triage.Evidence = field(default_factory=triage.Evidence)
 
     @property
     def labels(self) -> dict[str, str]:
@@ -43,19 +45,31 @@ async def latest_job(
     return await session.scalar(stmt)
 
 
-async def gene_terms_for(session: AsyncSession, genes: set[str]) -> dict[str, set[str]]:
-    """gene symbol -> the HPO terms annotated to it."""
+async def ontology_for(
+    session: AsyncSession, genes: set[str], case_terms: Sequence[str]
+) -> triage.Ontology:
+    """The HPO reference data this case needs: each gene's terms, and each case term's specificity.
+
+    Only the case's own terms need an information content: they are the denominator of the
+    phenotype score, and a gene's other annotations never enter it.
+    """
     if not genes:
-        return {}
+        return triage.Ontology()
     rows = await session.execute(
         select(GenePhenotype.gene_symbol, GenePhenotype.hpo_id).where(
             GenePhenotype.gene_symbol.in_(genes)
         )
     )
-    index: dict[str, set[str]] = {}
+    gene_terms: dict[str, set[str]] = {}
     for gene, hpo_id in rows:
-        index.setdefault(gene, set()).add(hpo_id)
-    return index
+        gene_terms.setdefault(gene, set()).add(hpo_id)
+    ic: dict[str, float] = {}
+    if case_terms:
+        weights = await session.execute(
+            select(HpoTerm.hpo_id, HpoTerm.ic).where(HpoTerm.hpo_id.in_(case_terms))
+        )
+        ic = {hpo_id: float(value) for hpo_id, value in weights}
+    return triage.Ontology(gene_terms=gene_terms, ic=ic)
 
 
 async def build(session: AsyncSession, case: Case) -> CaseView:
@@ -70,10 +84,14 @@ async def build(session: AsyncSession, case: Case) -> CaseView:
         )
     ).all()
     case_terms = [p.hpo_id for p in case.phenotypes]
-    gene_terms = await gene_terms_for(session, {v.gene for v in variants if v.gene})
+    ontology = await ontology_for(session, {v.gene for v in variants if v.gene}, case_terms)
+    evidence = triage.Evidence(
+        frequencies=job.has_frequencies, effect_scores=job.has_effect_scores
+    )
     return CaseView(
         case=case,
         job=job,
-        funnel=triage.funnel(variants, case_terms, gene_terms),
-        candidates=triage.rank(variants, case_terms, gene_terms),
+        funnel=triage.funnel(variants, case_terms, ontology, evidence),
+        candidates=triage.rank(variants, case_terms, ontology, evidence),
+        evidence=evidence,
     )

api/app/services/scoring.py

@@ -19,8 +19,9 @@ from sqlalchemy.ext.asyncio import AsyncSession
 from app.config import settings
 from app.models import Prediction, Variant
 
-# Must match rarelens_ml.features.RAW_COLUMNS.
-RAW_COLUMNS = ["impact", "consequence", "gnomad_af", "cadd_phred", "am_pathogenicity"]
+# Must match rarelens_ml.features.RAW_COLUMNS. Allele frequency is not among them: the ranking
+# scores frequency itself, and feeding it here too counted one measurement twice.
+RAW_COLUMNS = ["impact", "consequence", "cadd_phred", "am_pathogenicity"]
 CHUNK_SIZE = 5000
 
 _models: dict[str, Any] = {}  # model version -> loaded pyfunc
@@ -53,7 +54,6 @@ def raw_frame(variants: Sequence[Variant]) -> pd.DataFrame:
         {
             "impact": [v.impact for v in variants],
             "consequence": [v.consequence for v in variants],
-            "gnomad_af": [v.gnomad_af if v.gnomad_af is not None else float("nan") for v in variants],
             "cadd_phred": [v.annotations.get("CADD_PHRED") for v in variants],
             "am_pathogenicity": [v.annotations.get("am_pathogenicity") for v in variants],
         },

api/app/services/triage.py

@@ -1,14 +1,27 @@
 """Narrow a case's variants the way a clinical scientist does, and say why.
 
-The rank is a weighted sum of four parts a reviewer can audit. ClinVar is deliberately not one of
-them: it is shown beside the result as independent confirmation, so a variant never ranks highly
-merely because ClinVar already called it pathogenic.
+The rank is a weighted mean of four lines of evidence a reviewer can audit. ClinVar is deliberately
+not one of them: it is shown beside the result as independent confirmation, so a variant never
+ranks highly merely because ClinVar already called it pathogenic.
 
 Rarity and consequence *filter* (the usual first pass); phenotype only *ranks*, because a real
 diagnosis can sit in a gene nobody has annotated yet and filtering on it would hide exactly that.
+
+Two rules keep the number honest:
+
+**A line of evidence that was never looked up abstains.** It does not score zero, and it certainly
+does not score full marks. Treating "no gnomAD frequency in the annotation run" as "absent from
+gnomAD, therefore maximally rare" awarded every variant a free 0.25, which is a guess wearing the
+costume of a measurement. `Evidence` says what the run actually produced, and the weights
+renormalise over whatever is left, so the score stays on a 0-1 scale and means the same thing.
+
+**Each line of evidence is counted once.** The model used to take allele frequency as a feature
+while `rarity` scored the same frequency again, so roughly 45% of the rank was one measurement
+double-counted. The model no longer sees frequency (see rarelens_ml.features); it earns its weight
+only when it has something the other three do not already say, which means CADD or AlphaMissense.
 """
 from collections.abc import Mapping, Sequence
-from dataclasses import dataclass
+from dataclasses import dataclass, field
 
 from app.models import Variant
 
@@ -19,6 +32,49 @@ CANDIDATE_IMPACTS = frozenset({"HIGH", "MODERATE"})
 IMPACT_SEVERITY = {"HIGH": 1.0, "MODERATE": 0.6, "LOW": 0.2, "MODIFIER": 0.0}
 # Allele frequency ceiling -> score, rarest first.
 RARITY_STEPS = ((0.0, 1.0), (0.0001, 0.8), (0.001, 0.5), (0.01, 0.2))
+# A term HPO has never annotated to any gene cannot match anything, so its information content is
+# unknown. Treating it as maximally specific keeps it in the denominator and depresses every gene
+# equally, which is the neutral choice.
+DEFAULT_IC = 10.0
+
+
+@dataclass(frozen=True)
+class Evidence:
+    """What the annotation run actually produced, and therefore which components may score.
+
+    Decided once per job rather than per variant: components must be in play for every variant in
+    a case, or two variants would be scored against different denominators and their ranks would
+    not be comparable.
+    """
+
+    frequencies: bool = False       # did the run look up allele frequencies at all?
+    effect_scores: bool = False     # CADD / AlphaMissense, the only features the model adds
+
+    @property
+    def missing(self) -> list[str]:
+        absent = []
+        if not self.frequencies:
+            absent.append("rarity")
+        if not self.effect_scores:
+            absent.append("model")
+        return absent
+
+
+@dataclass(frozen=True)
+class Ontology:
+    """HPO reference data: what each gene is annotated with, and how specific each term is.
+
+    `gene_terms` is expected to be propagated up the ontology by scripts/load-hpo.py, so a case
+    term matches a gene annotated with any of its descendants. `ic` is information content,
+    -ln(fraction of genes carrying the term): "Bifid uvula" is worth many times "Abnormality of
+    the head", which nearly every gene in the corpus carries.
+    """
+
+    gene_terms: Mapping[str, set[str]] = field(default_factory=dict)
+    ic: Mapping[str, float] = field(default_factory=dict)
+
+    def weight(self, term: str) -> float:
+        return self.ic.get(term, DEFAULT_IC)
 
 
 @dataclass(frozen=True)
@@ -29,19 +85,21 @@ class Funnel:
     rare: int
     candidates: int
     phenotype_matched: int
+    frequencies: bool = False  # False means the "rare" step filtered nothing, because it could not
 
 
 @dataclass(frozen=True)
 class Candidate:
     variant: Variant
     score: float
-    components: dict[str, float]
+    components: dict[str, float | None]  # None: this evidence was not available
     matched_terms: list[str]
     scored: bool
 
 
 def rarity_score(af: float | None) -> float:
-    if af is None:  # absent from gnomAD
+    """Only meaningful when frequencies were annotated; None then means absent from gnomAD."""
+    if af is None:
         return 1.0
     for ceiling, score in RARITY_STEPS:
         if af <= ceiling:
@@ -54,14 +112,22 @@ def consequence_score(impact: str | None) -> float:
 
 
 def phenotype_score(
-    gene: str | None, case_terms: Sequence[str], gene_terms: Mapping[str, set[str]]
+    gene: str | None, case_terms: Sequence[str], ontology: Ontology
 ) -> tuple[float, list[str]]:
-    """What fraction of the patient's terms HPO associates with this gene, and which ones."""
+    """How much of the patient's phenotype HPO associates with this gene, weighted by specificity.
+
+    Information-content-weighted recall: the share of the *total specificity* of the patient's
+    terms that this gene accounts for. Plain term counting let a common term like global
+    developmental delay count as much as a near-pathognomonic one.
+    """
     if not gene or not case_terms:
         return 0.0, []
-    annotated = gene_terms.get(gene, set())
+    annotated = ontology.gene_terms.get(gene, set())
     matched = [term for term in case_terms if term in annotated]
-    return len(matched) / len(case_terms), matched
+    total = sum(ontology.weight(term) for term in case_terms)
+    if total <= 0:
+        return 0.0, matched
+    return sum(ontology.weight(term) for term in matched) / total, matched
 
 
 def is_rare(variant: Variant) -> bool:
@@ -72,35 +138,51 @@ def is_candidate(variant: Variant) -> bool:
     return is_rare(variant) and variant.impact in CANDIDATE_IMPACTS
 
 
-def funnel(
-    variants: Sequence[Variant], case_terms: Sequence[str], gene_terms: Mapping[str, set[str]]
-) -> Funnel:
+def combine(components: Mapping[str, float | None]) -> float:
+    """Weighted mean over the components that have evidence, renormalised to 0-1."""
+    weight = sum(WEIGHTS[name] for name, value in components.items() if value is not None)
+    if weight <= 0:
+        return 0.0
+    return sum(WEIGHTS[name] * value for name, value in components.items() if value is not None) / weight
+
+
+def weights_in_use(evidence: Evidence) -> dict[str, float]:
+    """The weights as actually applied, so the UI never shows a bar the score did not use."""
+    live = {name: w for name, w in WEIGHTS.items() if name not in evidence.missing}
+    total = sum(live.values())
+    return {name: round(w / total, 4) for name, w in live.items()} if total else {}
+
+
+def funnel(variants: Sequence[Variant], case_terms: Sequence[str], ontology: Ontology,
+           evidence: Evidence) -> Funnel:
     rare = [v for v in variants if is_rare(v)]
     candidates = [v for v in rare if v.impact in CANDIDATE_IMPACTS]
-    matched = sum(1 for v in candidates if phenotype_score(v.gene, case_terms, gene_terms)[1])
-    return Funnel(len(variants), len(rare), len(candidates), matched)
+    matched = sum(1 for v in candidates if phenotype_score(v.gene, case_terms, ontology)[1])
+    return Funnel(len(variants), len(rare), len(candidates), matched, evidence.frequencies)
 
 
 def evaluate(
-    variant: Variant, case_terms: Sequence[str], gene_terms: Mapping[str, set[str]]
+    variant: Variant, case_terms: Sequence[str], ontology: Ontology, evidence: Evidence
 ) -> Candidate:
     """Score one variant, whether or not it survived the filters."""
-    phenotype, matched = phenotype_score(variant.gene, case_terms, gene_terms)
+    phenotype, matched = phenotype_score(variant.gene, case_terms, ontology)
     prediction = variant.prediction
-    components = {
+    model: float | None = None
+    if evidence.effect_scores and prediction is not None:
+        model = float(prediction.score)
+    components: dict[str, float | None] = {
         "phenotype": phenotype,
-        "rarity": rarity_score(variant.gnomad_af),
+        "rarity": rarity_score(variant.gnomad_af) if evidence.frequencies else None,
         "consequence": consequence_score(variant.impact),
-        "model": float(prediction.score) if prediction is not None else 0.0,
+        "model": model,
     }
-    score = sum(WEIGHTS[name] * value for name, value in components.items())
-    return Candidate(variant, score, components, matched, prediction is not None)
+    return Candidate(variant, combine(components), components, matched, prediction is not None)
 
 
 def rank(
-    variants: Sequence[Variant], case_terms: Sequence[str], gene_terms: Mapping[str, set[str]]
+    variants: Sequence[Variant], case_terms: Sequence[str], ontology: Ontology, evidence: Evidence
 ) -> list[Candidate]:
-    candidates = [evaluate(v, case_terms, gene_terms) for v in variants if is_candidate(v)]
+    candidates = [evaluate(v, case_terms, ontology, evidence) for v in variants if is_candidate(v)]
     # id breaks ties, so equal scores do not shuffle between requests.
     candidates.sort(key=lambda c: (-c.score, c.variant.id))
     return candidates