P6 validated against GTCtoVCF too: zero unexplained genotype differences

- GTCtoVCF (Apache-2.0, 36 GSA samples, merged sites): 23.66 M identical calls; every difference named
  (intensity-only probes merged by the oracle, its opposite-strand indel bug EDGE-012, normalisation-based
  grouping, OUT-018 ploidy); indel representations identical, including 144 gtc2vcf cannot resolve
- validate_p6_vcf.py: rule-based categories (--bpm for [I/D]/[D/I] detection)
- docs: validation P6 complete, README status

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv

README.md

@@ -16,8 +16,13 @@ Phase **P5 (QC)** is complete: control-probe metrics, thresholds/profiles, two-a
 KING-robust relatedness, Mendelian errors, locus summary and a self-contained HTML report (`simurg qc report`). On 2141
 1000 Genomes samples all 717 pedigree parent–offspring pairs are recovered and trio Mendelian error rates are ~0.03 %
 (`docs/validation/P5.md`).
+Phase **P6 (outputs)** is complete: `simurg gtc-to-vcf` (plus-strand harmonisation with audit, merged multi-probe
+sites, per-sample VCF.gz + tabix or a cohort BCF + CSI) and `simurg export plink|bedgraph|finalreport|locus-summary`.
+On the GSA demo, genotypes match gtc2vcf and GTCtoVCF with zero unexplained differences and PLINK files match plink2
+(`docs/validation/P6.md`).
 Available commands: `simurg version`, `simurg inspect <files…> [--json]`, `simurg ingest`, `simurg normalize`, `simurg genotype`,
-`simurg qc controls|samples|contamination|relatedness|loci|report|suggest-thresholds`.
+`simurg qc controls|samples|contamination|relatedness|loci|report|suggest-thresholds`, `simurg gtc-to-vcf`,
+`simurg export plink|bedgraph|finalreport|locus-summary`.
 
 ## Quick start
 
@@ -28,6 +33,8 @@ bash scripts/bootstrap_refs.sh            # reference open-source repos → refs
 bash scripts/fetch_data.sh p0             # ~1.4 GB of public test data → data/
 bash scripts/make_oracle_gtcs.sh          # builds gtc2vcf, writes oracle GTCs
 bash scripts/make_oracle_qc.sh            # locus-summary oracle (BeadArrayFiles, separate venv)
+bash scripts/fetch_data.sh p6             # GRCh38 FASTA (~0.9 GB download, 3.1 GB unpacked)
+bash scripts/make_oracle_vcf.sh           # gtc2vcf oracle VCF for the P6 tests
 uv run pytest tests/oracle                # real-data acceptance tests (add -m "not slow" to skip the long ones)
 uv run simurg inspect data/p0/gsa_v4_pgx/idats_Demo_36/idats_Demo_36/207895980019/207895980019_R03C01_Grn.idat
 ```

docs/validation/P6.md

@@ -31,20 +31,37 @@ Both sides left-aligned with `bcftools norm -f`, compared per probe and sample a
 |---|---|---|
 | identical calls | 23 656 010 | |
 | identical no-calls | 102 077 | |
-| our merged multi-allelic site differs from the single-probe call | 2 414 | e.g. an Infinium II [G/T] probe calls GG while another probe at the site shows G/C: C shares G's dye, so the probe cannot see it — our site genotype is consistent with every probe |
+| `grouping_differs` | 6 424 | the probe belongs to a SimurgArray multi-allelic site (kept merged even with `--unsquash-duplicates`, OUT-034); the site genotype uses every probe (e.g. an Infinium II [G/T] probe calls GG while the site is G/C: C shares G's dye) or is missing when the probes contradict each other |
-| our merged site missing, probe called | 3 945 | the probes at the site contradict each other (`W_MERGE_INCONSISTENT`) or leave several genotypes (`W_MERGE_AMBIGUOUS`) |
+| `ours_missing_het_in_haploid` | 16 | male X non-PAR and MT heterozygous calls → missing (OUT-018) |
-| our merged site called, probe no-call | 65 | other probes at the site resolve it |
+| `ours_missing_haploid_Y` | 9 | Y of female samples → missing (OUT-018) |
-| het in a haploid context → missing (OUT-018) | 14 | male X non-PAR and MT heterozygous calls |
+| `representation_diff` | 5 184 (144 probes) | indels gtc2vcf leaves unresolved (`REF=I/D`); our representation agrees with GTCtoVCF (below) |
-| female Y → missing (OUT-018) | 11 | |
+| **unexplained** | **0** | |
-| indels gtc2vcf leaves unresolved (`REF=I/D`) that we resolve | 144 probes | checked against GTCtoVCF below |
+
-| **unexplained differences** | **0** | |
+Summary: `P6_gsa36_vs_gtc2vcf.json`.
 
 Locus sets: gtc2vcf writes 22 400 intensity-only loci that SimurgArray skips by design (OUT-019), and 49 indels it
 cannot resolve either (`W_SKIP_INDEL_REF` for us). Nothing we write is missing from gtc2vcf.
 
-### Genotypes vs GTCtoVCF (Apache-2.0), 36 samples, merged sites
+### Genotypes vs GTCtoVCF (Apache-2.0), 36 samples, merged sites (default squashing)
 
-(filled in below)
+GTCtoVCF 1.2.1 run under Python 3 with two local compatibility patches (K-022), alt-contig loci filtered.
+
+| Category | Count (probe × sample) | Explanation |
+|---|---|---|
+| identical calls | 23 656 575 | |
+| identical no-calls | 109 760 | |
+| `oracle_site_includes_probes_we_skip` | 2 917 | GTCtoVCF merges intensity-only CNV probes (e.g. `ilmnseq_CYP2C8_…_CNV`) into SNV sites; their alleles make its call ambiguous. SimurgArray skips intensity-only loci (OUT-019) |
+| `oracle_opposite_strand_indel_bug` | 177 | `[I/D]` and `[D/I]` duplicate indel probes all heterozygous: we write `0/1`, GTCtoVCF 1.2.1 `./.` — the reference bug named in OUT-020/EDGE-012 |
+| `grouping_differs` | 155 | indel probes designed 1–3 bp apart normalise to the same variant: we merge them (OUT-030), GTCtoVCF groups by manifest position; the probes disagree, so the merged site is missing |
+| `ours_missing_het_in_haploid` | 17 | OUT-018 |
+| `ours_missing_haploid_Y` | 9 | OUT-018 |
+| `oracle_diploid_candidates_in_haploid_context` | 2 | tri-allelic male-X site: we enumerate haploid candidates (OUT-031 step 3) and get one call; GTCtoVCF always enumerates diploid ones |
+| representation differences | 0 | every indel (including the 144 gtc2vcf cannot resolve) is written identically after left-alignment |
+| **unexplained** | **0** | |
+
+Probe sets: GTCtoVCF also writes the 22 400 intensity-only loci; it drops the 3 CYP2D6 indels whose `SourceSeq`
+has the dash on the right (patched to skip instead of crash), which SimurgArray writes. Summary:
+`P6_gsa36_vs_gtctovcf.json`.
 
 ### PLINK vs plink2 (GPL, subprocess oracle, built from source)
 

scripts/validate_p6_vcf.py

@@ -49,14 +49,27 @@ def main() -> int:
     ap.add_argument("--bcftools", required=True)
     ap.add_argument("--out", required=True, type=Path)
     ap.add_argument("--name", default="oracle")
+    ap.add_argument("--bpm", type=Path, help="manifest, to recognise opposite-strand indel designs (EDGE-012)")
     a = ap.parse_args()
 
     samples = subprocess.run([a.bcftools, "query", "-l", str(a.oracle)], capture_output=True, text=True,
                              check=True).stdout.split()  # fmt: skip
     oracle: dict[str, tuple[str, str, str, str, list[str]]] = {}
+    oracle_group: dict[str, frozenset[str]] = {}
     for f in query(a.bcftools, a.fasta, a.oracle):
-        for pid in f[0].replace(";", ",").split(","):
+        ids = f[0].replace(";", ",").split(",")
+        for pid in ids:
             oracle[pid] = (f[1], f[2], f[3], f[4], f[5:])
+            oracle_group[pid] = frozenset(ids)
+    snp: dict[str, str] = {}
+    if a.bpm:
+        sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "src"))
+        from simurg.formats.bpm import read_bpm
+
+        m = read_bpm(a.bpm)
+        snp = dict(zip(m.names, m.snps, strict=True))
+    first = a.ours / f"{samples[0]}.snv.vcf.gz"
+    ours_all = {pid for f in query(a.bcftools, a.fasta, first) for pid in f[0].replace(";", ",").split(",")}
     cats: Counter[str] = Counter()
     examples: dict[str, list[Any]] = defaultdict(list)
     ours_ids: set[str] = set()
@@ -71,6 +84,8 @@ def main() -> int:
             merged = len(ids) > 1
             ours_al = alleles_of(f[5], f[3], f[4])
             haploid = "/" not in f[5] and f[5] != "."
+            group = frozenset(ids)
+            mixed_indel = {snp.get(x) for x in ids} >= {"[I/D]", "[D/I]"}
             for pid in ids:
                 if si == 0:
                     ours_ids.add(pid)
@@ -88,17 +103,27 @@ def main() -> int:
                 if ours_al == theirs:
                     cats["match_called" if ours_al else "match_missing"] += 1
                     continue
-                if ours_al is None:
+                og = oracle_group[pid]
-                    if merged:
+                if og - ours_all:
-                        key = "ours_missing_merged_site"
+                    key = "oracle_site_includes_probes_we_skip"  # e.g. intensity-only CNV probes (OUT-019)
+                elif og != group and (merged or len(og) > 1):
+                    key = "grouping_differs"  # same normalised variant, different manifest positions (OUT-030)
+                elif ours_al is not None and theirs is None and mixed_indel:
+                    key = "oracle_opposite_strand_indel_bug"  # EDGE-012: [I/D] + [D/I] both het → 0/1
+                elif ours_al is None:
+                    if theirs is not None and theirs[0] != theirs[1] and f[5] == ".":
+                        key = "ours_missing_het_in_haploid"  # OUT-018
                     elif f[1] in ("Y", "chrY") and f[5] == ".":
-                        key = "ours_missing_haploid_Y"
+                        key = "ours_missing_haploid_Y"  # OUT-018: Y of female samples
-                    elif theirs is not None and theirs[0] != theirs[1] and (f[5] == "." or haploid):
+                    elif merged:
-                        key = "ours_missing_het_in_haploid"
+                        key = "ours_missing_merged_site"
                     else:
                         key = "ours_missing_UNEXPLAINED"
                 elif theirs is None:
-                    key = "oracle_missing_ours_called_merged" if merged else "oracle_missing_ours_called"
+                    if merged and haploid:
+                        key = "oracle_diploid_candidates_in_haploid_context"  # OUT-031 enumerates with the ploidy
+                    else:
+                        key = "oracle_missing_ours_called_merged" if merged else "oracle_missing_ours_called"
                 else:
                     key = "diff_merged_site" if merged else "diff_UNEXPLAINED"
                 cats[key] += 1