The number that changed what the model is allowed to do

Held-out evaluation with whole genes held out, never single variants (Grimm et al. 2015): 312,025 training and 74,239 test variants across 7,728 and 1,932 genes, with no gene on both sides.

Model

AUROC

AUPRC

missense AUROC

missense AUPRC

v2 — with gnomAD allele frequency as a feature

0.986

0.954

0.872

0.725

v3 — allele frequency removed

0.966

0.881

0.500

0.398

0.500

AUROC on missense variants once frequency is removed. Exactly random.

What that proves

Strip frequency out and the model cannot tell one missense variant from another at all — nothing is left but the consequence class, so every missense row scores identically. The respectable-looking 0.872 was never variant-effect knowledge. It was allele frequency.

And the frequency feature was circular

ACMG's BA1/BS1 criteria assign ClinVar's benign labels using allele frequency. The feature had partly caused the label, so the model was rediscovering the rule that produced its own training data.

The consequence for the product

Frequency is no longer a feature — the ranking already scores it explicitly and auditably, and feeding it to the model as well put ~45% of every rank on one measurement counted twice. The model now abstains unless it has CADD or AlphaMissense, because 0.500 is the measurement saying it has nothing else to add.