kemal
6d22b5e19b
360 arrays vs NYGC 30x SV: all samples OK (K-027 gate fix), PASS precision DEL 44 % / DUP 57 % (first run 39 % / 34 % with filters ignored), 61 % without the one excess_events array; rare DEL sensitivity 30 %; LOH replicated 85 %. m7g.4xlarge spot, 17 min. docs/validation/V2_cn.md, K-027.
116 lines
7.6 KiB
Markdown
116 lines
7.6 KiB
Markdown
# v2 module validation — genome-wide CNV / LOH / mosaicism (`simurg cn`) — interim
|
||
|
||
Date: 2026-09-29. Status: **not accepted yet** — constitutional calling works, mosaic calling produces mostly false
|
||
positives on the real-data set; see "Next steps". Spec: CYTO-001 … CYTO-040.
|
||
|
||
## Delivered
|
||
|
||
`simurg cn build-model` (GC 1 kb / 100 kb / 1 Mb, PFB from the cluster file, sex-aware batch reference from normal
|
||
GTCs), `simurg cn call` (GC-wave correction, smoothing, HMM with constitutional, gain-LOH, mosaic and haploid
|
||
states, events, post-filters, thresholds, centromere mask, CNV VCF + tabix, segments, summary, sidecar),
|
||
`simurg cn annotate` (JSON report, ISCN 2020, cytobands, RefSeq genes), `scripts/build_annotation_bundle.sh`.
|
||
Unit tests: `tests/unit/test_cn.py` (11).
|
||
|
||
## Synthetic implants (CYTO-022), GSA v4 demo, 22 synthetic samples, 216 events
|
||
|
||
Reference model from 30 samples, events implanted into the other 6 (`scripts/v2/cn_synthetic_eval.py`,
|
||
`V2_cn_synthetic_gsa36.json`), 4 replicates per cell:
|
||
|
||
| Event | 1 Mb | 5 Mb | 20 Mb |
|
||
|---|---|---|---|
|
||
| constitutional DEL / DUP | 4/4 · 2/4 | 4/4 · 4/4 | 4/4 · 4/4 |
|
||
| constitutional LOH (threshold 3 Mb) | 0/4 | 3/4 | 4/4 |
|
||
| mosaic loss / gain / CN-LOH, 15 % | 3/4 · 1/4 · 0/4 | **4/4 · 4/4 · 4/4** | 4/4 · 4/4 · 4/4 |
|
||
| mosaic, 5–10 % | mostly missed | mostly missed | 0–3/4 |
|
||
|
||
The CYTO-022 gate (≥ 90 % at 15 % / 5 Mb) is met on implanted data (12/12, small n).
|
||
|
||
## Real data: GSE165845 (360 GSA-MD arrays, 120 individuals × 3) vs NYGC 30x SV callset (freeze V3)
|
||
|
||
Cloud run 2026-09-29 (m7g.4xlarge, `cn call` 360 samples in 7 min 40 s; reference from 120 arrays of the same set).
|
||
Truth: non-reference DEL/DUP ≥ 25 kb of the same individual, ≥ 10 array probes; match = reciprocal overlap ≥ 50 %.
|
||
|
||
* Every sample was flagged `SampleQuality` (summary `logr_dev` median 0.246 — the GSA v1 C1 cluster file does not
|
||
match the GSA-MD chips, P8) and 46 fell into whole-chromosome-only mode, although the LRR used for calling has a
|
||
robust SD of 0.099 after the batch reference. The gate now uses the SD of the normalised LRR (K-027). The numbers
|
||
below ignore `SampleQuality` (`--ignore-filter`), evaluated from the run's sidecar.
|
||
|
||
| QUAL ≥ 20 | DEL | DUP |
|
||
|---|---|---|
|
||
| sensitivity 25–50 / 50–100 / 100–500 / 500–1000 kb | 7 / 8 / 8 / 7 % | 1 / 3 / 11 / 13 % |
|
||
| precision (all calls of the class) | 39 % | 34 % |
|
||
|
||
| Call type (QUAL ≥ 20) | calls | in NYGC truth | seen in another replicate |
|
||
|---|---|---|---|
|
||
| constitutional DEL | 227 | **77 %** | 50 % |
|
||
| constitutional DUP | 23 | **74 %** | 30 % |
|
||
| LOH | — | — | 74 % |
|
||
| mosaic loss | 399 | 12 % | 12 % |
|
||
| mosaic gain | 567 | 23 % | 23 % |
|
||
|
||
Constitutional calls are mostly true; the mosaic states generate most calls and most false positives (large
|
||
BAF-supported "mosaic" events are 0–2 % in truth and rarely reproduced: shifted heterozygous BAF bands of the
|
||
mismatched cluster file mimic allelic imbalance). Small BAF-supported mosaic gains are often real germline
|
||
duplications labelled mosaic (43 % in truth).
|
||
|
||
## Pi tuning rounds (2026-09-29), 40 arrays (20 individuals × replicates 1, 2), reference = replicate 3 of all 120
|
||
|
||
`scripts/v2/cn_tune.sh TAG [options]`; truth as above, QUAL ≥ 20, PASS only (after the K-027 noise-gate fix).
|
||
|
||
| Round | DEL calls / in truth | DUP | mosaic loss PASS / in truth | mosaic gain PASS / in truth | LOH replicated |
|
||
|---|---|---|---|---|---|
|
||
| base (per-probe reference het BAF spread in the emissions) | 22 / 100 % | 3 / 67 % | 29 / 11 % | 24 / 38 % | 80 % |
|
||
| `--mosaic-prior 0.0005` | 25 / 96 % | 4 / 75 % | 27 / 12 % | 22 / 36 % | 80 % |
|
||
| `--mosaic-prior 0.0005 --switch-distance 1e8` | 24 / 96 % | 4 / 75 % | 25 / 14 % | 23 / 35 % | 80 % |
|
||
| CYTO-041 (noise-normalised BAF split, complete-loss promotion) | 27 / 93 % | 4 / 75 % | 19 / 5 % | 19 / 42 % | 80 % |
|
||
| + CYTO-042 (per-probe LRR noise from the reference; **default**) | 20 / **100 %** | 4 / **100 %** | **11** / 9 % | **13** / **62 %** | 80 % |
|
||
|
||
* The mosaic prior and switch distance barely matter. CYTO-041 moves heterozygote-free "mosaic" losses (compressed
|
||
constitutional deletions) to DEL (`HIGHFRACTION`) and sends noise-driven mosaic calls to `LowFraction`
|
||
(MDEL 29 → 19 PASS, MDUP 24 → 19). On the implanted set (`data/v2cn/syn3`) MF estimates stay unbiased
|
||
(15 % → 0.16, 20 % → 0.21, 30 % → 0.30), detection is unchanged (126/216) and the CYTO-022 gate still passes.
|
||
* Remaining mosaic-loss false positives are long, low-amplitude LRR dips (−0.05 … −0.1) with a heterozygous BAF
|
||
spread only 1.0–1.3 × the probe's reference spread, or short events with < 20 heterozygous probes (MF from LRR).
|
||
Without phasing a < 20 % mosaic loss cannot be separated from regional BAF noise (K-027).
|
||
* CYTO-042 (per-probe emission SD = sample SD × the probe's reference LRR spread / median spread, clipped to
|
||
[0.5, 3]) halves the mosaic calls and makes constitutional calls fully precise, at the cost of 5 rare deletions
|
||
(rare DEL 24/64 → 19/64; rare DUP 10/68 → 12/68). Synthetic set with the rebuilt model (`data/v2cn/syn4`): calls
|
||
564 → 402, detection 126 → 124/216, the CYTO-022 gate (15 % / 5 Mb: 4/4 loss, gain, CN-LOH) still passes.
|
||
Disable with `--no-probe-lrr-sd`.
|
||
* **Sensitivity is limited by common CNVs.** Detected truth DEL by carrier frequency among the 120 individuals:
|
||
< 5 %: 24/64 (38 %), 5–20 %: 1/20, 20–50 %: 1/80, > 50 %: 0/156 (DUP 10/66, 0/2, 1/24, 0/40). The batch
|
||
reference is the per-probe median LRR, so an event carried by most reference samples is normalised away (and the
|
||
cluster file, trained on the population, already absorbs part of it). Array CN calling targets rare events; the
|
||
headline sensitivity figures above are dominated by common polymorphic deletions.
|
||
|
||
## Final full-cohort run (cloud, 2026-09-29, commit ccf7f4f, CYTO-041/042 defaults)
|
||
|
||
The run used m7g.4xlarge spot: 17 min in total (`cn build-model` 1 min 17 s, 6.0 GB; `cn call` of 360 samples 7 min 44 s, 1.7 GB).
|
||
Truth and matching are the same as above. PASS calls only, QUAL ≥ 20, no filters ignored
|
||
(`V2_cn_gse165845_final.json`).
|
||
|
||
* **Sample QC.** All 360 samples are `OK`; the first cloud run flagged every sample `SampleQuality` (the K-027 gate
|
||
fix). 14 samples carry `high_noise` and one carries `excess_events`.
|
||
* **The `excess_events` array.** 203610890020_R06C01 (NA19923) has 238 DEL and 688 mosaic calls, and none of its
|
||
147 CNV calls is in truth. It is a bad array, and it is the only one flagged.
|
||
* **Per-sample call counts (median).** 1 DEL, 0 DUP, 4 mosaic.
|
||
|
||
| | DEL | DUP |
|
||
|---|---|---|
|
||
| precision, PASS calls | 213/480 = **44 %** (first run, filters ignored: 39 %) | 142/249 = **57 %** (first run, filters ignored: 34 %) |
|
||
| precision, CNV calls without R06C01 | 355/582 = **61 %** (DEL + DUP) | |
|
||
| sensitivity 25–50 / 50–100 / 100–500 / 500–1000 kb | 5 / 7 / 8 / 9 % | 0 / 3 / 10 / 14 % |
|
||
| sensitivity by carrier frequency < 5 % / 5–20 % / ≥ 20 % | **30 %** / 5 % / 0.05 % | 13 % / 21 % / 0.6 % |
|
||
|
||
* **Replicate reproducibility.** CNV 35 %, LOH 85 %.
|
||
* **What CYTO-041/042 changed.** Precision is clearly higher and sensitivity is unchanged. The low overall
|
||
sensitivity is the common-CNV effect described above, not missed rare events.
|
||
* **Rare deletions.** 30 % of rare (< 5 %) truth deletions are found, the same as in the Pi tuning.
|
||
|
||
## Next steps
|
||
|
||
1. Mosaic losses below ~20 %: phasing-based BAF (MoChA-style, MIT) is the principled fix.
|
||
2. Rare-event sensitivity (currently DEL 30 %, DUP 18 % for events ≥ 25 kb with ≥ 10 probes): exclude reference
|
||
samples carrying the event when computing the per-probe median (iterative reference), size-aware transitions.
|
||
3. ~~A final full-cohort run~~ — done (section above).
|