V2 CN: CYTO-042 results (precision up, mosaic calls halved); pi_run reports process memory without page cache

docs/validation/V2_cn.md

@@ -62,7 +62,8 @@ duplications labelled mosaic (43 % in truth).
 | base (per-probe reference het BAF spread in the emissions) | 22 / 100 % | 3 / 67 % | 29 / 11 % | 24 / 38 % | 80 % |
 | `--mosaic-prior 0.0005` | 25 / 96 % | 4 / 75 % | 27 / 12 % | 22 / 36 % | 80 % |
 | `--mosaic-prior 0.0005 --switch-distance 1e8` | 24 / 96 % | 4 / 75 % | 25 / 14 % | 23 / 35 % | 80 % |
-| CYTO-041 (noise-normalised BAF split, complete-loss promotion) | 27 / 93 % | 4 / 75 % | **19** / 5 % | **19** / 42 % | 80 % |
+| CYTO-041 (noise-normalised BAF split, complete-loss promotion) | 27 / 93 % | 4 / 75 % | 19 / 5 % | 19 / 42 % | 80 % |
+| + CYTO-042 (per-probe LRR noise from the reference; **default**) | 20 / **100 %** | 4 / **100 %** | **11** / 9 % | **13** / **62 %** | 80 % |
 
 * The mosaic prior and switch distance barely matter. CYTO-041 moves heterozygote-free "mosaic" losses (compressed
   constitutional deletions) to DEL (`HIGHFRACTION`) and sends noise-driven mosaic calls to `LowFraction`
@@ -71,6 +72,11 @@ duplications labelled mosaic (43 % in truth).
 * Remaining mosaic-loss false positives are long, low-amplitude LRR dips (−0.05 … −0.1) with a heterozygous BAF
   spread only 1.0–1.3 × the probe's reference spread, or short events with < 20 heterozygous probes (MF from LRR).
   Without phasing a < 20 % mosaic loss cannot be separated from regional BAF noise (K-027).
+* CYTO-042 (per-probe emission SD = sample SD × the probe's reference LRR spread / median spread, clipped to
+  [0.5, 3]) halves the mosaic calls and makes constitutional calls fully precise, at the cost of 5 rare deletions
+  (rare DEL 24/64 → 19/64; rare DUP 10/68 → 12/68). Synthetic set with the rebuilt model (`data/v2cn/syn4`): calls
+  564 → 402, detection 126 → 124/216, the CYTO-022 gate (15 % / 5 Mb: 4/4 loss, gain, CN-LOH) still passes.
+  Disable with `--no-probe-lrr-sd`.
 * **Sensitivity is limited by common CNVs.** Detected truth DEL by carrier frequency among the 120 individuals:
   < 5 %: 24/64 (38 %), 5–20 %: 1/20, 20–50 %: 1/80, > 50 %: 0/156 (DUP 10/66, 0/2, 1/24, 0/40). The batch
   reference is the per-probe median LRR, so an event carried by most reference samples is normalised away (and the
@@ -79,8 +85,7 @@ duplications labelled mosaic (43 % in truth).
 
 ## Next steps
 
-1. Report sensitivity for rare events (carrier frequency < 5 %) as the primary metric in `cn_validate_truth.py`.
+1. Mosaic losses below ~20 %: phasing-based BAF (MoChA-style, MIT) is the principled fix.
-2. Mosaic losses: phasing-based BAF (MoChA-style) is the principled fix for low fractions; until then consider a
+2. Rare-event sensitivity (currently DEL 30 %, DUP 18 % for events ≥ 25 kb with ≥ 10 probes): exclude reference
-   minimum heterozygous-probe count for mosaic PASS calls.
+   samples carrying the event when computing the per-probe median (iterative reference), size-aware transitions.
-3. Sensitivity for 25–500 kb events: per-probe LRR SD from the reference.
+3. A final full-cohort run (cloud, ~$0.10, needs approval).
-4. A final full-cohort run (cloud, ~$0.10, needs approval) once the above settle.

scripts/pi_run.sh

@@ -21,10 +21,13 @@ else
 fi
 pid=$!
 peak=0
+anon_peak=0  # process memory without page cache (memory.peak counts cached file pages, e.g. a FASTA being read)
 tripped=0
 while kill -0 "$pid" 2>/dev/null; do
   if [[ -r $cg/memory.peak ]]; then
     peak=$(( $(cat "$cg/memory.peak") / 1024 ))
+    anon=$(awk '$1 == "anon" {print int($2 / 1024)}' "$cg/memory.stat" 2>/dev/null || echo 0)
+    (( anon > anon_peak )) && anon_peak=$anon
   else
     rss=$(ps -o rss= -g "$pid" 2>/dev/null | awk '{s+=$1} END {print s+0}'); (( rss > peak )) && peak=$rss
   fi
@@ -41,5 +44,7 @@ wait "$pid"; rc=$?
 (( tripped )) && rc=137
 (( rc == 137 && ! tripped )) && [[ $mode == cgroup* ]] \
   && echo "pi_run: the job was killed (SIGKILL); most likely it exceeded MemoryMax=$MEM" >&2
-echo "pi_run: exit $rc, peak memory $((peak / 1024)) MiB ($mode, guard ${MIN_FREE_MB} MiB free)" >&2
+anon_note=""
+(( anon_peak > 0 )) && anon_note=", process $((anon_peak / 1024)) MiB without page cache"
+echo "pi_run: exit $rc, peak memory $((peak / 1024)) MiB$anon_note ($mode, guard ${MIN_FREE_MB} MiB free)" >&2
 exit $rc