diff --git a/docs/known_unknowns.md b/docs/known_unknowns.md index 6f2d917..1eda868 100644 --- a/docs/known_unknowns.md +++ b/docs/known_unknowns.md @@ -4,6 +4,7 @@ Plan §20 lists the open questions; this file records what the implementation ha | ID | Date | Topic | Finding | Status | |---|---|---|---|---| +| K-027 | 2026-09-29 | CN/LOH (v2) | CYTO-014 noise gate: the summary `logr_dev` also measures cluster-file mismatch (GSE165845: 0.246, all samples `SampleQuality`) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. | Open: mosaic calling needs rework (`docs/validation/V2_cn.md`). | | K-026 | 2026-09-29 | Methylation QC (v2) | Spec deviations: vendor EPIC manifests put a 5th field on some control lines (≥ 4 fields accepted); detection p follows sesame's pOOBAH (1 − max(F(M), F(U))) rather than the ambiguous "1 − ECDF(M + U)"; minfi-style `Sentrix_ID`/`Sentrix_Position` sheet keys accepted. Against sesame 1.28.1 on minfiDataEPIC: β r = 0.99998 (max \|Δβ\| 0.011), detection calls 99.1 % identical, noob signals within 0.5 % (median) / 5 % (max). The systematic remainder is unexplained; it is not the negative controls (sesame did not attach the supplied control table). | Open; `docs/validation/V2_methylation.md`. Next step: dump sesame's background vectors and noob parameters to find the difference. | | K-025 | 2026-09-28 | Scaling limits (P9) | With the Rust kernels a Graviton3 core genotypes 78–80 GSA samples/min. At > 8 workers a 125 MB/s gp3 volume caps the run at ~620 samples/min (GTC ≈ 13 MB each); on a RAM disk efficiency is 0.77 at 16 and 0.48 at 32 workers for 360 samples, explained by a fixed start-up cost (≈ 2.5 s + 0.2 s per worker). Linear scaling for large batches is inferred from that model, not measured. | `docs/validation/P9.md`; measure a ≥ 5000-sample batch when one is at hand. | | K-024 | 2026-09-28 | GSE165845 (P8) | GSA-MD v1 chips have no public cluster file; the GSA v1 C1 EGT with the C2 build-38 BPM covers 99.996 % of bead addresses and gives median LogR deviation 0.246 (cluster mismatch, not noise). GSM5051309 (NA20362 rep 2) is a failed array (call rate 0.762, concordance 0.336 with its own replicates). 27 510 array records have no NYGC 30x record and are scored as hom-ref; 1444 of them are systematically called alt. | Recorded in `docs/validation/P8.md`; results reported with and without these. | diff --git a/docs/validation/V2_cn.md b/docs/validation/V2_cn.md new file mode 100644 index 0000000..df69fff --- /dev/null +++ b/docs/validation/V2_cn.md @@ -0,0 +1,63 @@ +# v2 module validation — genome-wide CNV / LOH / mosaicism (`simurg cn`) — interim + +Date: 2026-09-29. Status: **not accepted yet** — constitutional calling works, mosaic calling produces mostly false +positives on the real-data set; see "Next steps". Spec: CYTO-001 … CYTO-040. + +## Delivered + +`simurg cn build-model` (GC 1 kb / 100 kb / 1 Mb, PFB from the cluster file, sex-aware batch reference from normal +GTCs), `simurg cn call` (GC-wave correction, smoothing, HMM with constitutional, gain-LOH, mosaic and haploid +states, events, post-filters, thresholds, centromere mask, CNV VCF + tabix, segments, summary, sidecar), +`simurg cn annotate` (JSON report, ISCN 2020, cytobands, RefSeq genes), `scripts/build_annotation_bundle.sh`. +Unit tests: `tests/unit/test_cn.py` (11). + +## Synthetic implants (CYTO-022), GSA v4 demo, 22 synthetic samples, 216 events + +Reference model from 30 samples, events implanted into the other 6 (`scripts/v2/cn_synthetic_eval.py`, +`V2_cn_synthetic_gsa36.json`), 4 replicates per cell: + +| Event | 1 Mb | 5 Mb | 20 Mb | +|---|---|---|---| +| constitutional DEL / DUP | 4/4 · 2/4 | 4/4 · 4/4 | 4/4 · 4/4 | +| constitutional LOH (threshold 3 Mb) | 0/4 | 3/4 | 4/4 | +| mosaic loss / gain / CN-LOH, 15 % | 3/4 · 1/4 · 0/4 | **4/4 · 4/4 · 4/4** | 4/4 · 4/4 · 4/4 | +| mosaic, 5–10 % | mostly missed | mostly missed | 0–3/4 | + +The CYTO-022 gate (≥ 90 % at 15 % / 5 Mb) is met on implanted data (12/12, small n). + +## Real data: GSE165845 (360 GSA-MD arrays, 120 individuals × 3) vs NYGC 30x SV callset (freeze V3) + +Cloud run 2026-09-29 (m7g.4xlarge, `cn call` 360 samples in 7 min 40 s; reference from 120 arrays of the same set). +Truth: non-reference DEL/DUP ≥ 25 kb of the same individual, ≥ 10 array probes; match = reciprocal overlap ≥ 50 %. + +* Every sample was flagged `SampleQuality` (summary `logr_dev` median 0.246 — the GSA v1 C1 cluster file does not + match the GSA-MD chips, P8) and 46 fell into whole-chromosome-only mode, although the LRR used for calling has a + robust SD of 0.099 after the batch reference. The gate now uses the SD of the normalised LRR (K-027). The numbers + below ignore `SampleQuality` (`--ignore-filter`), evaluated from the run's sidecar. + +| QUAL ≥ 20 | DEL | DUP | +|---|---|---| +| sensitivity 25–50 / 50–100 / 100–500 / 500–1000 kb | 7 / 8 / 8 / 7 % | 1 / 3 / 11 / 13 % | +| precision (all calls of the class) | 39 % | 34 % | + +| Call type (QUAL ≥ 20) | calls | in NYGC truth | seen in another replicate | +|---|---|---|---| +| constitutional DEL | 227 | **77 %** | 50 % | +| constitutional DUP | 23 | **74 %** | 30 % | +| LOH | — | — | 74 % | +| mosaic loss | 399 | 12 % | 12 % | +| mosaic gain | 567 | 23 % | 23 % | + +Constitutional calls are mostly true; the mosaic states generate most calls and most false positives (large +BAF-supported "mosaic" events are 0–2 % in truth and rarely reproduced: shifted heterozygous BAF bands of the +mismatched cluster file mimic allelic imbalance). Small BAF-supported mosaic gains are often real germline +duplications labelled mosaic (43 % in truth). + +## Next steps + +1. Mosaic calling: require consistent BAF imbalance with a per-probe reference of heterozygous BAF spread, raise the + evidence for mosaic states (prior/size), and relabel compressed germline gains/losses as constitutional; tune on + GSE165845 and re-measure (replicate reproducibility is the cheapest check). +2. Sensitivity for 25–500 kb events: lower switch cost for short events with strong LRR, per-probe LRR SD from the + reference. +3. Re-run the truth comparison; document the final numbers. diff --git a/docs/validation/V2_cn_gse165845_q10.json b/docs/validation/V2_cn_gse165845_q10.json new file mode 100644 index 0000000..be85fe8 --- /dev/null +++ b/docs/validation/V2_cn_gse165845_q10.json @@ -0,0 +1,89 @@ +{ + "samples": 360, + "min_qual": 10.0, + "ignored_filters": [ + "SampleQuality" + ], + "sensitivity": { + "DEL": { + "100-500kb": { + "found": 230, + "truth": 1554, + "rate": 0.148 + }, + "1000-infkb": { + "found": 0, + "truth": 15, + "rate": 0.0 + }, + "25-50kb": { + "found": 54, + "truth": 225, + "rate": 0.24 + }, + "50-100kb": { + "found": 132, + "truth": 894, + "rate": 0.1477 + }, + "500-1000kb": { + "found": 51, + "truth": 180, + "rate": 0.2833 + } + }, + "DUP": { + "100-500kb": { + "found": 279, + "truth": 1059, + "rate": 0.2635 + }, + "1000-infkb": { + "found": 9, + "truth": 21, + "rate": 0.4286 + }, + "25-50kb": { + "found": 4, + "truth": 198, + "rate": 0.0202 + }, + "50-100kb": { + "found": 55, + "truth": 462, + "rate": 0.119 + }, + "500-1000kb": { + "found": 19, + "truth": 78, + "rate": 0.2436 + } + } + }, + "precision": { + "DEL": { + "in_truth": 473, + "calls": 1814, + "rate": 0.2607 + }, + "DUP": { + "in_truth": 367, + "calls": 1494, + "rate": 0.2456 + } + }, + "replicate_reproducibility": { + "CNV": { + "matched": 2132, + "comparisons": 6616, + "rate": 0.3222 + }, + "LOH": { + "matched": 400, + "comparisons": 532, + "rate": 0.7519 + } + }, + "calls_per_sample_median": 6.0, + "loh_calls_per_sample_median": 0.0 +} \ No newline at end of file diff --git a/docs/validation/V2_cn_gse165845_q20.json b/docs/validation/V2_cn_gse165845_q20.json new file mode 100644 index 0000000..7489b43 --- /dev/null +++ b/docs/validation/V2_cn_gse165845_q20.json @@ -0,0 +1,89 @@ +{ + "samples": 360, + "min_qual": 20.0, + "ignored_filters": [ + "SampleQuality" + ], + "sensitivity": { + "DEL": { + "100-500kb": { + "found": 126, + "truth": 1554, + "rate": 0.0811 + }, + "1000-infkb": { + "found": 0, + "truth": 15, + "rate": 0.0 + }, + "25-50kb": { + "found": 16, + "truth": 225, + "rate": 0.0711 + }, + "50-100kb": { + "found": 70, + "truth": 894, + "rate": 0.0783 + }, + "500-1000kb": { + "found": 13, + "truth": 180, + "rate": 0.0722 + } + }, + "DUP": { + "100-500kb": { + "found": 117, + "truth": 1059, + "rate": 0.1105 + }, + "1000-infkb": { + "found": 6, + "truth": 21, + "rate": 0.2857 + }, + "25-50kb": { + "found": 1, + "truth": 198, + "rate": 0.0051 + }, + "50-100kb": { + "found": 15, + "truth": 462, + "rate": 0.0325 + }, + "500-1000kb": { + "found": 10, + "truth": 78, + "rate": 0.1282 + } + } + }, + "precision": { + "DEL": { + "in_truth": 222, + "calls": 571, + "rate": 0.3888 + }, + "DUP": { + "in_truth": 149, + "calls": 436, + "rate": 0.3417 + } + }, + "replicate_reproducibility": { + "CNV": { + "matched": 554, + "comparisons": 2014, + "rate": 0.2751 + }, + "LOH": { + "matched": 286, + "comparisons": 394, + "rate": 0.7259 + } + }, + "calls_per_sample_median": 2.0, + "loh_calls_per_sample_median": 0.0 +} \ No newline at end of file diff --git a/scripts/v2/cn_validate_truth.py b/scripts/v2/cn_validate_truth.py index 0ddaf2b..d6429c3 100644 --- a/scripts/v2/cn_validate_truth.py +++ b/scripts/v2/cn_validate_truth.py @@ -47,6 +47,7 @@ def main() -> int: ap.add_argument("--truth", required=True, type=Path) ap.add_argument("--bpm", required=True) ap.add_argument("--min-qual", type=float, default=20) + ap.add_argument("--ignore-filter", action="append", default=[], help="FILTER values not to exclude (repeatable)") ap.add_argument("--out", required=True, type=Path) a = ap.parse_args() a.out.mkdir(parents=True, exist_ok=True) @@ -66,7 +67,8 @@ def main() -> int: for s in side["samples"]: if s.get("status") != "OK": continue - calls[s["sample_id"]] = [e for e in s["events"] if e["quality"] >= a.min_qual and e["filters"] == ["PASS"]] + ign = set(a.ignore_filter) | {"PASS"} + calls[s["sample_id"]] = [e for e in s["events"] if e["quality"] >= a.min_qual and set(e["filters"]) <= ign] truth: dict[str, list[tuple[str, int, int, str]]] = defaultdict(list) with a.truth.open() as fh: for line in fh: @@ -142,6 +144,7 @@ def main() -> int: summary = { "samples": len(per_sample), "min_qual": a.min_qual, + "ignored_filters": sorted(a.ignore_filter), "sensitivity": { t: {b: {"found": v[0], "truth": v[1], "rate": rate(v)} for b, v in sorted(d.items())} for t, d in sens.items() diff --git a/src/simurg/pipeline/cn.py b/src/simurg/pipeline/cn.py index 1cb5b18..6b05a22 100644 --- a/src/simurg/pipeline/cn.py +++ b/src/simurg/pipeline/cn.py @@ -323,8 +323,11 @@ def call_sample(si: SampleInput, ctx: dict[str, Any], opts: CnCallOptions) -> di lrr, gcfit = gc_correct(lrr, model.gc, auto) lrr = lrr - float(np.nanmedian(lrr[auto])) logr_dev = si.logr_dev if math.isfinite(si.logr_dev) else float(np.nanstd(lrr[auto])) - noisy = logr_dev > 0.2 - whole_chrom_only = logr_dev > 0.3 + # CYTO-014 gates on the noise of the LRR the caller actually uses (after the batch reference and GC correction): + # the summary's logr_dev also measures cluster-file mismatch, which the reference removes (K-027) + gate_sd = float(np.nanstd(lrr[auto])) if model.ref_lrr is not None else logr_dev + noisy = gate_sd > 0.2 + whole_chrom_only = gate_sd > 0.3 if noisy: flags.append("high_noise") if whole_chrom_only: @@ -417,6 +420,7 @@ def call_sample(si: SampleInput, ctx: dict[str, Any], opts: CnCallOptions) -> di if excess: row["qc_flags"] = ";".join([*flags, "excess_events:" + ",".join(excess)]) row["lrr_sd_segmentation"] = seg_sd + row["lrr_sd_gate"] = gate_sd row["baf_sd"] = baf_sd # CYTO-033 sample and chromosome statistics (unsmoothed LRR), for cn annotate la = lrr[auto & np.isfinite(lrr)]