ArrayTrain: BIC-selected unbounded θ mixtures + HWE-guided labelling

Fixed θ windows mislabelled compressed/asymmetric loci (2-cluster loci: 1418 vs
186 errors). Now 1/2/3-component mixtures chosen by BIC, labelled by HWE
goodness of fit plus weak position and gap priors, split clusters merged.
GSE165845 chr22 vs 1000G 30x (240 arrays): concordance 99.44 % (v1 99.27 %,
vendor EGT 99.46 %), NRD 1.30 % (1.89 %, 1.23 %), call rate 99.61 %.
docs/validation/V2_arraytrain.md, K-030; plan §12, CLI-038.

docs/known_unknowns.md

@@ -4,6 +4,7 @@ Plan §20 lists the open questions; this file records what the implementation ha
 
 | ID | Date | Topic | Finding | Status |
 |---|---|---|---|---|
+| K-030 | 2026-09-29 | ArrayTrain (v2) | The clustering is ours, not the vendor's (plan §12): unbounded 1/2/3-component θ mixtures chosen by BIC, with components labelled by HWE goodness of fit plus weak position and gap priors. Fixed θ windows (v1) mislabelled compressed and asymmetric loci. On GSE165845 (train on replicate 3, evaluate on replicates 1–2, chr22 vs 1000G 30x) the result is 99.44 % concordance, 1.30 % NRD and 99.61 % call rate, against 99.46 % / 1.23 % / 99.15 % with the vendor EGT. The parameters were tuned on this evaluation (a flat optimum). GenTrain and cluster-separation scores are ours, so GenCall scores are not comparable with vendor-file scores. | Open: second cohort/chip; score calibration (`docs/validation/V2_arraytrain.md`). |
 | K-029 | 2026-09-29 | PGx star alleles (v2) | (1) PGX-012 error model: ε(GS) = 0.5·(1 − GS)² was replaced by ε = 1e-3 (GS ≥ 0.15) / 0.05 — after probe masking 0 of 9 740 GSA v4 PGx calls disagreed with 1000G 30x at any GS, and the old model made correct low-GS calls lose to the prior (spec amended). (2) PGX-010 masks derived from 14 individuals (spec: ≥ 100); a variant absent from the filtered 1000G panel counts as hom-ref only if rare (CYP2B6 rs2279343 is filtered out, not absent). (3) CPIC core-allele definitions use IUPAC wildcards; alleles without definitions (NUDT15 *2, structural CYP2D6) are not callable from SNVs. (4) GeT-RM uses legacy UGT1A1/SLCO1B1 names. Result vs GeT-RM (16 CEPH): 100/109 gene calls, CYP2D6/2C19/2C9 41/45. (5) DPWG: no public machine-readable genotype→phenotype mapping; CPIC mappings are used for genes CPIC covers (as ClinPGx does), DPWG-only genes (CYP1A2, CYP3A4, F5) get no phenotype. (6) GeT-RM definitions are older than CPIC/PharmVar for UGT1A1, SLCO1B1, CYP1A2 and CYP2C8 (e.g. PharmVar CYP2C8 *19 = rs2071426 alone). Result vs GeT-RM (16 CEPH, 21 genes): 154/168 comparable gene calls. Independent check on GSE165845 (GSA v1 C2, 120 individuals × 3, WGS-derived truth, 2-fold masks): 95.7 % array-resolvable concordance, 67.9 % top-solution (the chip lacks many defining variants), replicate concordance 99.2 % without the failed array. | Open: KNMP DPWG tables if they become available (`docs/validation/V2_pgx_star.md`). |
 | K-028 | 2026-09-29 | PGx CN (v2) | Deviations from PCN-009: the per-probe reference is the training cohort's median stored in the model (a calling-batch median is not CN 2 where deletions are common, e.g. GSTM1 null ≈ 50 %); no GC-correction step; the CN mixture is ours: µ_c = δ + γ·log2((c+1)/3) per region with a free lower CN-0 component and Hardy–Weinberg-constrained weights (probe compression differs by region: γ ≈ 0.19 CYP2D6.p5 … 0.56 GSTT1). Interim validation on the GSA v4 PGx demo — whose 36 arrays turned out to be 30 CEPH/1000G individuals — against NYGC 30x SV RD_CN: 224/240 region calls (GSTM1, UGT2B17 30/30), but in-sample (trained on the same arrays, < 96 samples, priors chosen with this truth in view). | Open: validate on an independent GSA-PGx cohort; CYP2D6 via GeT-RM star alleles (`docs/validation/V2_pgx_cn.md`). |
 | K-027 | 2026-09-29 | CN/LOH (v2) | CYTO-014 noise gate: the summary `logr_dev` also measures cluster-file mismatch (GSE165845: 0.246, all samples `SampleQuality`) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. Pi tuning: mosaic prior / switch distance barely matter; CYTO-041 (noise-normalised BAF split, complete-loss promotion) cuts PASS mosaic calls by ~30 % without losing implanted ≥ 15 % mosaics. Two limits remain: (1) without phasing, < 20 % mosaic losses are not separable from regional BAF noise; (2) common CNVs (carrier frequency > 20 %) are almost never detected (1/236 DEL) because the median-LRR batch reference absorbs them; rare DEL (< 5 %) 38 %. | Open: rare-event sensitivity as primary metric; phasing-based mosaic BAF (`docs/validation/V2_cn.md`). |

docs/validation/V2_arraytrain.md

@@ -0,0 +1,86 @@
+# V2 — ArrayTrain (plan §12)
+
+`simurg train` builds an EGT cluster file from a cohort's GTCs. It does not reproduce the vendor's clustering, so the
+benchmark is genotype accuracy against sequencing truth: an EGT trained by SimurgArray versus the vendor's published
+cluster file, used by the same GenCall emulation on the same arrays.
+
+## Set-up
+
+- **Data.** GSE165845: 120 1000 Genomes individuals, each on 3 GSA v1 arrays. The C2 build-38 BPM is used; the
+  public GSA v1 C1 cluster file is the vendor baseline (K-024).
+- **Training.** Replicate 3, 120 GTCs (59 F, 60 M, 1 U), all 618 540 loci. Pi 5, 69 min, peak 1.9 GB.
+- **Evaluation.** Replicates 1–2 (240 arrays) go from IDAT through `simurg genotype` with the trained EGT, then
+  `simurg gtc-to-vcf`. The same arrays are also genotyped with the vendor EGT.
+- **Truth.** The NYGC 30x phased 1000G panel on chr22: 8278 array SNV sites (`scripts/train/concordance_chr.py`).
+  Concordance counts called genotypes only. NRD is the non-reference discordance.
+
+The evaluation individuals are also in the training set, but on different arrays. The cluster file never sees their
+truth; the same holds for the vendor file.
+
+## Results (240 arrays, chr22)
+
+| Cluster file | Call rate | Concordance | NRD |
+|---|---|---|---|
+| ArrayTrain v1 (fixed θ windows) | 99.47 % | 99.27 % | 1.89 % |
+| **ArrayTrain v2 (BIC + HWE labelling)** | **99.61 %** | **99.44 %** | **1.30 %** |
+| Vendor GSA v1 C1 EGT | 99.15 % | 99.46 % | 1.23 % |
+
+`V2_arraytrain_gse165845.json` holds the numbers. v2 calls 0.46 % more genotypes than the vendor file at nearly the
+same concordance (−0.02 points).
+
+## What changed from v1, and why
+
+The v1 training fitted one three-component θ mixture. Its means were bounded to fixed windows (AA ≤ 0.4,
+0.2 ≤ AB ≤ 0.8, BB ≥ 0.6), and close clusters were merged by a fixed θ gap.
+
+A per-locus diagnosis on chr22 compared our trained clusters, the vendor clusters and the truth for 80 arrays:
+
+- **Errors concentrate in few loci.** The top 20 loci held 43 % of the errors.
+- **2-cluster loci were the problem.** There, v1 made 1418 errors against the vendor's 186.
+- **The cause was the fixed θ windows.** Many loci are compressed or asymmetric in θ; for example the vendor file
+  has AA 0.05 / AB 0.27 / BB 0.45, or AA 0.37 / AB 0.68 / BB 0.96. With fixed windows such loci were mislabelled:
+  GSA-rs2003814 (truth 14 AB + 66 BB) was clustered as AA + AB.
+
+v2 (`src/simurg/train/clusters.py`) replaces this with:
+
+1. **Unbounded 1-, 2- and 3-component mixtures** on θ, plus a uniform outlier component. Each is fitted by EM from
+   several starts: canonical positions and θ quantiles.
+   - An SD floor of 0.02 inside the EM stops a component collapsing onto the θ = 0 / 1 spikes.
+   - BIC chooses among admissible fits. A fit is admissible if every component has ≥ max(3, 1 %) of the samples and
+     adjacent components are ≥ 1.5 standardised gaps apart.
+2. **Labelling of the θ-ordered components** by the best non-decreasing map onto AA/AB/BB. A map's score adds up:
+   - the HWE goodness of fit of the implied genotype counts, with an inbreeding allowance F = 0.1. This is the fit
+     relative to the saturated multinomial, so monomorphic labellings are not favoured automatically;
+   - weak priors on the cluster positions: AA ~ N(0.03, 0.2), AB ~ N(0.5, 0.25), BB ~ N(0.97, 0.2);
+   - a weak prior on the θ gaps between genotype groups: 0.42 ± 0.12 per step.
+
+   Adjacent components may share a genotype (a split cluster) only when their standardised gap is < 3.5.
+
+Fast chr22 loop (train on replicate 3, call 80 evaluation arrays, score against truth):
+
+| Step | Errors (80 arrays) |
+|---|---|
+| v1 | 2562 |
+| Unbounded mixtures + HWE labelling of 2-cluster fits | 1793 |
+| + MIN_SEP 1.5 and position prior | 1611 |
+| + general labelling with split-cluster merges (HWE goodness of fit) | 1380 |
+| Vendor EGT | 1237 |
+
+- **Parameter sweep.** MERGE_SEP {2.5, 3.5, 5} × HWE F {0.05, 0.1, 0.2} × gap SD {0.12, 0.2}. Every setting
+  lands at 99.70–99.79 % accuracy; MERGE_SEP 5 is clearly worse. The chosen values are in the middle of a flat
+  optimum.
+- **Where the errors are now.** By our cluster count, v2's errors are 123 at 1-cluster loci (vendor 157), 217 at
+  2-cluster loci (vendor 113) and 1040 at 3-cluster loci (vendor 967).
+- **Many 3-cluster errors are shared with the vendor.** Both files see three clear clusters where the truth is
+  monomorphic, e.g. rs191067607 and rs5758036. These are probe or truth problems, not clustering errors.
+
+## Limits (K-030)
+
+- Only one cohort and one chromosome were used for accuracy. The tuning used this evaluation, so the numbers are
+  optimistic for other chips and cohorts.
+- 120 training arrays is close to the 100-sample minimum. Rare-allele clusters are imputed from cross-locus
+  relations, not observed.
+- The trained EGT scores (GenTrain, cluster separation) are ours. GenCall scores from a trained file are not
+  comparable with scores from vendor files. At a matched call rate, GenCall score ranks v1's errors worse than the
+  vendor's.
+- Training takes 69 min for 120 × 618 540 on the Pi (8 EM runs per locus); a Rust kernel would help at cohort scale.

docs/validation/V2_arraytrain_gse165845.json

@@ -0,0 +1,28 @@
+{
+ "dataset": "GSE165845 GSA v1 (C2 build-38 BPM); train = replicate 3 (120 arrays), evaluate = replicates 1-2 (240 arrays)",
+ "truth": "1000G NYGC 30x phased panel, chr22 SNVs on the array",
+ "egt_v1_fixed_windows": {
+  "chrom": "22",
+  "samples": 240,
+  "truth_sites": 8278,
+  "call_rate": 0.9947053434807118,
+  "concordance": 0.9926765907658587,
+  "nrd": 0.018946074088334416
+ },
+ "egt_v2_bic_hwe": {
+  "chrom": "22",
+  "samples": 240,
+  "truth_sites": 8278,
+  "call_rate": 0.9960623540307643,
+  "concordance": 0.9943676222304063,
+  "nrd": 0.013034595450012981
+ },
+ "vendor_gsa_v1_c1_egt": {
+  "chrom": "22",
+  "samples": 240,
+  "truth_sites": 8278,
+  "call_rate": 0.9915201940887493,
+  "concordance": 0.9945781643910259,
+  "nrd": 0.012251034586575862
+ }
+}