v2 CN: real-data validation vs NYGC 30x SVs (interim), noise gate on normalised LRR (K-027)

GSE165845 360 arrays: constitutional DEL/DUP calls 77/74 % in the NYGC SV truth, LOH reproduced across replicates 74 %; mosaic calls are mostly false positives (12-23 %), sensitivity for 25 kb-1 Mb events 7-13 % at QUAL >= 20. All samples had been flagged SampleQuality by the summary logr_dev (cluster-file mismatch); the gate now uses the SD of the LRR after the batch reference. Validation script: --ignore-filter.
2026-09-29 07:42:06 +01:00
1 parent 726fe045f5
commit a5728d84e3
@@ -4,6 +4,7 @@ Plan §20 lists the open questions; this file records what the implementation ha

 | ID | Date | Topic | Finding | Status |
 |---|---|---|---|---|
+| K-027 | 2026-09-29 | CN/LOH (v2) | CYTO-014 noise gate: the summary `logr_dev` also measures cluster-file mismatch (GSE165845: 0.246, all samples `SampleQuality`) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. | Open: mosaic calling needs rework (`docs/validation/V2_cn.md`). |
 | K-026 | 2026-09-29 | Methylation QC (v2) | Spec deviations: vendor EPIC manifests put a 5th field on some control lines (≥ 4 fields accepted); detection p follows sesame's pOOBAH (1 − max(F(M), F(U))) rather than the ambiguous "1 − ECDF(M + U)"; minfi-style `Sentrix_ID`/`Sentrix_Position` sheet keys accepted. Against sesame 1.28.1 on minfiDataEPIC: β r = 0.99998 (max \|Δβ\| 0.011), detection calls 99.1 % identical, noob signals within 0.5 % (median) / 5 % (max). The systematic remainder is unexplained; it is not the negative controls (sesame did not attach the supplied control table). | Open; `docs/validation/V2_methylation.md`. Next step: dump sesame's background vectors and noob parameters to find the difference. |
 | K-025 | 2026-09-28 | Scaling limits (P9) | With the Rust kernels a Graviton3 core genotypes 78–80 GSA samples/min. At > 8 workers a 125 MB/s gp3 volume caps the run at ~620 samples/min (GTC ≈ 13 MB each); on a RAM disk efficiency is 0.77 at 16 and 0.48 at 32 workers for 360 samples, explained by a fixed start-up cost (≈ 2.5 s + 0.2 s per worker). Linear scaling for large batches is inferred from that model, not measured. | `docs/validation/P9.md`; measure a ≥ 5000-sample batch when one is at hand. |
 | K-024 | 2026-09-28 | GSE165845 (P8) | GSA-MD v1 chips have no public cluster file; the GSA v1 C1 EGT with the C2 build-38 BPM covers 99.996 % of bead addresses and gives median LogR deviation 0.246 (cluster mismatch, not noise). GSM5051309 (NA20362 rep 2) is a failed array (call rate 0.762, concordance 0.336 with its own replicates). 27 510 array records have no NYGC 30x record and are scored as hom-ref; 1444 of them are systematically called alt. | Recorded in `docs/validation/P8.md`; results reported with and without these. |
@@ -0,0 +1,63 @@
+# v2 module validation — genome-wide CNV / LOH / mosaicism (`simurg cn`) — interim
+
+Date: 2026-09-29. Status: **not accepted yet** — constitutional calling works, mosaic calling produces mostly false
+positives on the real-data set; see "Next steps". Spec: CYTO-001 … CYTO-040.
+
+## Delivered
+
+`simurg cn build-model` (GC 1 kb / 100 kb / 1 Mb, PFB from the cluster file, sex-aware batch reference from normal
+GTCs), `simurg cn call` (GC-wave correction, smoothing, HMM with constitutional, gain-LOH, mosaic and haploid
+states, events, post-filters, thresholds, centromere mask, CNV VCF + tabix, segments, summary, sidecar),
+`simurg cn annotate` (JSON report, ISCN 2020, cytobands, RefSeq genes), `scripts/build_annotation_bundle.sh`.
+Unit tests: `tests/unit/test_cn.py` (11).
+
+## Synthetic implants (CYTO-022), GSA v4 demo, 22 synthetic samples, 216 events
+
+Reference model from 30 samples, events implanted into the other 6 (`scripts/v2/cn_synthetic_eval.py`,
+`V2_cn_synthetic_gsa36.json`), 4 replicates per cell:
+
+| Event | 1 Mb | 5 Mb | 20 Mb |
+|---|---|---|---|
+| constitutional DEL / DUP | 4/4 · 2/4 | 4/4 · 4/4 | 4/4 · 4/4 |
+| constitutional LOH (threshold 3 Mb) | 0/4 | 3/4 | 4/4 |
+| mosaic loss / gain / CN-LOH, 15 % | 3/4 · 1/4 · 0/4 | **4/4 · 4/4 · 4/4** | 4/4 · 4/4 · 4/4 |
+| mosaic, 5–10 % | mostly missed | mostly missed | 0–3/4 |
+
+The CYTO-022 gate (≥ 90 % at 15 % / 5 Mb) is met on implanted data (12/12, small n).
+
+## Real data: GSE165845 (360 GSA-MD arrays, 120 individuals × 3) vs NYGC 30x SV callset (freeze V3)
+
+Cloud run 2026-09-29 (m7g.4xlarge, `cn call` 360 samples in 7 min 40 s; reference from 120 arrays of the same set).
+Truth: non-reference DEL/DUP ≥ 25 kb of the same individual, ≥ 10 array probes; match = reciprocal overlap ≥ 50 %.
+
+* Every sample was flagged `SampleQuality` (summary `logr_dev` median 0.246 — the GSA v1 C1 cluster file does not
+  match the GSA-MD chips, P8) and 46 fell into whole-chromosome-only mode, although the LRR used for calling has a
+  robust SD of 0.099 after the batch reference. The gate now uses the SD of the normalised LRR (K-027). The numbers
+  below ignore `SampleQuality` (`--ignore-filter`), evaluated from the run's sidecar.
+
+| QUAL ≥ 20 | DEL | DUP |
+|---|---|---|
+| sensitivity 25–50 / 50–100 / 100–500 / 500–1000 kb | 7 / 8 / 8 / 7 % | 1 / 3 / 11 / 13 % |
+| precision (all calls of the class) | 39 % | 34 % |
+
+| Call type (QUAL ≥ 20) | calls | in NYGC truth | seen in another replicate |
+|---|---|---|---|
+| constitutional DEL | 227 | **77 %** | 50 % |
+| constitutional DUP | 23 | **74 %** | 30 % |
+| LOH | — | — | 74 % |
+| mosaic loss | 399 | 12 % | 12 % |
+| mosaic gain | 567 | 23 % | 23 % |
+
+Constitutional calls are mostly true; the mosaic states generate most calls and most false positives (large
+BAF-supported "mosaic" events are 0–2 % in truth and rarely reproduced: shifted heterozygous BAF bands of the
+mismatched cluster file mimic allelic imbalance). Small BAF-supported mosaic gains are often real germline
+duplications labelled mosaic (43 % in truth).
+
+## Next steps
+
+1. Mosaic calling: require consistent BAF imbalance with a per-probe reference of heterozygous BAF spread, raise the
+   evidence for mosaic states (prior/size), and relabel compressed germline gains/losses as constitutional; tune on
+   GSE165845 and re-measure (replicate reproducibility is the cheapest check).
+2. Sensitivity for 25–500 kb events: lower switch cost for short events with strong LRR, per-probe LRR SD from the
+   reference.
+3. Re-run the truth comparison; document the final numbers.
@@ -0,0 +1,89 @@
+{
+ "samples": 360,
+ "min_qual": 10.0,
+ "ignored_filters": [
+  "SampleQuality"
+ ],
+ "sensitivity": {
+  "DEL": {
+   "100-500kb": {
+    "found": 230,
+    "truth": 1554,
+    "rate": 0.148
+   },
+   "1000-infkb": {
+    "found": 0,
+    "truth": 15,
+    "rate": 0.0
+   },
+   "25-50kb": {
+    "found": 54,
+    "truth": 225,
+    "rate": 0.24
+   },
+   "50-100kb": {
+    "found": 132,
+    "truth": 894,
+    "rate": 0.1477
+   },
+   "500-1000kb": {
+    "found": 51,
+    "truth": 180,
+    "rate": 0.2833
+   }
+  },
+  "DUP": {
+   "100-500kb": {
+    "found": 279,
+    "truth": 1059,
+    "rate": 0.2635
+   },
+   "1000-infkb": {
+    "found": 9,
+    "truth": 21,
+    "rate": 0.4286
+   },
+   "25-50kb": {
+    "found": 4,
+    "truth": 198,
+    "rate": 0.0202
+   },
+   "50-100kb": {
+    "found": 55,
+    "truth": 462,
+    "rate": 0.119
+   },
+   "500-1000kb": {
+    "found": 19,
+    "truth": 78,
+    "rate": 0.2436
+   }
+  }
+ },
+ "precision": {
+  "DEL": {
+   "in_truth": 473,
+   "calls": 1814,
+   "rate": 0.2607
+  },
+  "DUP": {
+   "in_truth": 367,
+   "calls": 1494,
+   "rate": 0.2456
+  }
+ },
+ "replicate_reproducibility": {
+  "CNV": {
+   "matched": 2132,
+   "comparisons": 6616,
+   "rate": 0.3222
+  },
+  "LOH": {
+   "matched": 400,
+   "comparisons": 532,
+   "rate": 0.7519
+  }
+ },
+ "calls_per_sample_median": 6.0,
+ "loh_calls_per_sample_median": 0.0
+}
@@ -0,0 +1,89 @@
+{
+ "samples": 360,
+ "min_qual": 20.0,
+ "ignored_filters": [
+  "SampleQuality"
+ ],
+ "sensitivity": {
+  "DEL": {
+   "100-500kb": {
+    "found": 126,
+    "truth": 1554,
+    "rate": 0.0811
+   },
+   "1000-infkb": {
+    "found": 0,
+    "truth": 15,
+    "rate": 0.0
+   },
+   "25-50kb": {
+    "found": 16,
+    "truth": 225,
+    "rate": 0.0711
+   },
+   "50-100kb": {
+    "found": 70,
+    "truth": 894,
+    "rate": 0.0783
+   },
+   "500-1000kb": {
+    "found": 13,
+    "truth": 180,
+    "rate": 0.0722
+   }
+  },
+  "DUP": {
+   "100-500kb": {
+    "found": 117,
+    "truth": 1059,
+    "rate": 0.1105
+   },
+   "1000-infkb": {
+    "found": 6,
+    "truth": 21,
+    "rate": 0.2857
+   },
+   "25-50kb": {
+    "found": 1,
+    "truth": 198,
+    "rate": 0.0051
+   },
+   "50-100kb": {
+    "found": 15,
+    "truth": 462,
+    "rate": 0.0325
+   },
+   "500-1000kb": {
+    "found": 10,
+    "truth": 78,
+    "rate": 0.1282
+   }
+  }
+ },
+ "precision": {
+  "DEL": {
+   "in_truth": 222,
+   "calls": 571,
+   "rate": 0.3888
+  },
+  "DUP": {
+   "in_truth": 149,
+   "calls": 436,
+   "rate": 0.3417
+  }
+ },
+ "replicate_reproducibility": {
+  "CNV": {
+   "matched": 554,
+   "comparisons": 2014,
+   "rate": 0.2751
+  },
+  "LOH": {
+   "matched": 286,
+   "comparisons": 394,
+   "rate": 0.7259
+  }
+ },
+ "calls_per_sample_median": 2.0,
+ "loh_calls_per_sample_median": 0.0
+}