V2 CN validation: Pi tuning rounds, CYTO-041 effect, common-CNV sensitivity limit (K-027)

docs/known_unknowns.md

@@ -4,7 +4,7 @@ Plan §20 lists the open questions; this file records what the implementation ha
 
 | ID | Date | Topic | Finding | Status |
 |---|---|---|---|---|
-| K-027 | 2026-09-29 | CN/LOH (v2) | CYTO-014 noise gate: the summary `logr_dev` also measures cluster-file mismatch (GSE165845: 0.246, all samples `SampleQuality`) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. | Open: mosaic calling needs rework (`docs/validation/V2_cn.md`). |
+| K-027 | 2026-09-29 | CN/LOH (v2) | CYTO-014 noise gate: the summary `logr_dev` also measures cluster-file mismatch (GSE165845: 0.246, all samples `SampleQuality`) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. Pi tuning: mosaic prior / switch distance barely matter; CYTO-041 (noise-normalised BAF split, complete-loss promotion) cuts PASS mosaic calls by ~30 % without losing implanted ≥ 15 % mosaics. Two limits remain: (1) without phasing, < 20 % mosaic losses are not separable from regional BAF noise; (2) common CNVs (carrier frequency > 20 %) are almost never detected (1/236 DEL) because the median-LRR batch reference absorbs them; rare DEL (< 5 %) 38 %. | Open: rare-event sensitivity as primary metric; phasing-based mosaic BAF (`docs/validation/V2_cn.md`). |
 | K-026 | 2026-09-29 | Methylation QC (v2) | Spec deviations: vendor EPIC manifests put a 5th field on some control lines (≥ 4 fields accepted); detection p follows sesame's pOOBAH (1 − max(F(M), F(U))) rather than the ambiguous "1 − ECDF(M + U)"; minfi-style `Sentrix_ID`/`Sentrix_Position` sheet keys accepted. Against sesame 1.28.1 on minfiDataEPIC: β r = 0.99998 (max \|Δβ\| 0.011), detection calls 99.1 % identical, noob signals within 0.5 % (median) / 5 % (max). The systematic remainder is unexplained; it is not the negative controls (sesame did not attach the supplied control table). | Open; `docs/validation/V2_methylation.md`. Next step: dump sesame's background vectors and noob parameters to find the difference. |
 | K-025 | 2026-09-28 | Scaling limits (P9) | With the Rust kernels a Graviton3 core genotypes 78–80 GSA samples/min. At > 8 workers a 125 MB/s gp3 volume caps the run at ~620 samples/min (GTC ≈ 13 MB each); on a RAM disk efficiency is 0.77 at 16 and 0.48 at 32 workers for 360 samples, explained by a fixed start-up cost (≈ 2.5 s + 0.2 s per worker). Linear scaling for large batches is inferred from that model, not measured. | `docs/validation/P9.md`; measure a ≥ 5000-sample batch when one is at hand. |
 | K-024 | 2026-09-28 | GSE165845 (P8) | GSA-MD v1 chips have no public cluster file; the GSA v1 C1 EGT with the C2 build-38 BPM covers 99.996 % of bead addresses and gives median LogR deviation 0.246 (cluster mismatch, not noise). GSM5051309 (NA20362 rep 2) is a failed array (call rate 0.762, concordance 0.336 with its own replicates). 27 510 array records have no NYGC 30x record and are scored as hom-ref; 1444 of them are systematically called alt. | Recorded in `docs/validation/P8.md`; results reported with and without these. |

docs/validation/V2_cn.md

@@ -53,11 +53,34 @@ BAF-supported "mosaic" events are 0–2 % in truth and rarely reproduced: shifte
 mismatched cluster file mimic allelic imbalance). Small BAF-supported mosaic gains are often real germline
 duplications labelled mosaic (43 % in truth).
 
+## Pi tuning rounds (2026-09-29), 40 arrays (20 individuals × replicates 1, 2), reference = replicate 3 of all 120
+
+`scripts/v2/cn_tune.sh TAG [options]`; truth as above, QUAL ≥ 20, PASS only (after the K-027 noise-gate fix).
+
+| Round | DEL calls / in truth | DUP | mosaic loss PASS / in truth | mosaic gain PASS / in truth | LOH replicated |
+|---|---|---|---|---|---|
+| base (per-probe reference het BAF spread in the emissions) | 22 / 100 % | 3 / 67 % | 29 / 11 % | 24 / 38 % | 80 % |
+| `--mosaic-prior 0.0005` | 25 / 96 % | 4 / 75 % | 27 / 12 % | 22 / 36 % | 80 % |
+| `--mosaic-prior 0.0005 --switch-distance 1e8` | 24 / 96 % | 4 / 75 % | 25 / 14 % | 23 / 35 % | 80 % |
+| CYTO-041 (noise-normalised BAF split, complete-loss promotion) | 27 / 93 % | 4 / 75 % | **19** / 5 % | **19** / 42 % | 80 % |
+
+* The mosaic prior and switch distance barely matter. CYTO-041 moves heterozygote-free "mosaic" losses (compressed
+  constitutional deletions) to DEL (`HIGHFRACTION`) and sends noise-driven mosaic calls to `LowFraction`
+  (MDEL 29 → 19 PASS, MDUP 24 → 19). On the implanted set (`data/v2cn/syn3`) MF estimates stay unbiased
+  (15 % → 0.16, 20 % → 0.21, 30 % → 0.30), detection is unchanged (126/216) and the CYTO-022 gate still passes.
+* Remaining mosaic-loss false positives are long, low-amplitude LRR dips (−0.05 … −0.1) with a heterozygous BAF
+  spread only 1.0–1.3 × the probe's reference spread, or short events with < 20 heterozygous probes (MF from LRR).
+  Without phasing a < 20 % mosaic loss cannot be separated from regional BAF noise (K-027).
+* **Sensitivity is limited by common CNVs.** Detected truth DEL by carrier frequency among the 120 individuals:
+  < 5 %: 24/64 (38 %), 5–20 %: 1/20, 20–50 %: 1/80, > 50 %: 0/156 (DUP 10/66, 0/2, 1/24, 0/40). The batch
+  reference is the per-probe median LRR, so an event carried by most reference samples is normalised away (and the
+  cluster file, trained on the population, already absorbs part of it). Array CN calling targets rare events; the
+  headline sensitivity figures above are dominated by common polymorphic deletions.
+
 ## Next steps
 
-1. Mosaic calling: require consistent BAF imbalance with a per-probe reference of heterozygous BAF spread, raise the
-   evidence for mosaic states (prior/size), and relabel compressed germline gains/losses as constitutional; tune on
-   GSE165845 and re-measure (replicate reproducibility is the cheapest check).
-2. Sensitivity for 25–500 kb events: lower switch cost for short events with strong LRR, per-probe LRR SD from the
-   reference.
-3. Re-run the truth comparison; document the final numbers.
+1. Report sensitivity for rare events (carrier frequency < 5 %) as the primary metric in `cn_validate_truth.py`.
+2. Mosaic losses: phasing-based BAF (MoChA-style) is the principled fix for low fractions; until then consider a
+   minimum heterozygous-probe count for mosaic PASS calls.
+3. Sensitivity for 25–500 kb events: per-probe LRR SD from the reference.
+4. A final full-cohort run (cloud, ~$0.10, needs approval) once the above settle.