PGx star alleles: pgx star-allele call (PGX-011..021), probe masks vs 1000G, GeT-RM validation (K-029)

- simurg/pgx/star.py: solver-free diplotype enumeration incl. CN forms (deletion, xN), wildcards, array-equivalence classes, raw_score/score with CPIC frequency priors; calibrated error model (spec PGX-012 amended). - pipeline/pgx_star.py: DB gene models, SNV + PGx CN VCF evidence, masks, diplotype_summary.tsv + pgx_calls.json. - scripts/pgx: derive_masks.py (array vs 1000G 30x over HTTPS, GS calibration), getrm_truth.py, validate_star_getrm.py. - GSA v4 PGx demo vs GeT-RM (16 CEPH): 100/109 gene calls; CYP2D6/2C19/2C9 41/45 (docs/validation/V2_pgx_star.md).
2026-09-29 14:08:03 +01:00
1 parent 7e9ff28501
commit 194c087b24
@@ -4,6 +4,7 @@ Plan §20 lists the open questions; this file records what the implementation ha

 | ID | Date | Topic | Finding | Status |
 |---|---|---|---|---|
+| K-029 | 2026-09-29 | PGx star alleles (v2) | (1) PGX-012 error model: ε(GS) = 0.5·(1 − GS)² was replaced by ε = 1e-3 (GS ≥ 0.15) / 0.05 — after probe masking 0 of 9 740 GSA v4 PGx calls disagreed with 1000G 30x at any GS, and the old model made correct low-GS calls lose to the prior (spec amended). (2) PGX-010 masks derived from 14 individuals (spec: ≥ 100); a variant absent from the filtered 1000G panel counts as hom-ref only if rare (CYP2B6 rs2279343 is filtered out, not absent). (3) CPIC core-allele definitions use IUPAC wildcards; alleles without definitions (NUDT15 *2, structural CYP2D6) are not callable from SNVs. (4) GeT-RM uses legacy UGT1A1/SLCO1B1 names. Result vs GeT-RM (16 CEPH): 100/109 gene calls, CYP2D6/2C19/2C9 41/45. | Open: hybrids, G6PD/variant-based genes, PharmVar genes, phenotypes; larger/independent truth (`docs/validation/V2_pgx_star.md`). |
 | K-028 | 2026-09-29 | PGx CN (v2) | Deviations from PCN-009: the per-probe reference is the training cohort's median stored in the model (a calling-batch median is not CN 2 where deletions are common, e.g. GSTM1 null ≈ 50 %); no GC-correction step; the CN mixture is ours: µ_c = δ + γ·log2((c+1)/3) per region with a free lower CN-0 component and Hardy–Weinberg-constrained weights (probe compression differs by region: γ ≈ 0.19 CYP2D6.p5 … 0.56 GSTT1). Interim validation on the GSA v4 PGx demo — whose 36 arrays turned out to be 30 CEPH/1000G individuals — against NYGC 30x SV RD_CN: 224/240 region calls (GSTM1, UGT2B17 30/30), but in-sample (trained on the same arrays, < 96 samples, priors chosen with this truth in view). | Open: validate on an independent GSA-PGx cohort; CYP2D6 via GeT-RM star alleles (`docs/validation/V2_pgx_cn.md`). |
 | K-027 | 2026-09-29 | CN/LOH (v2) | CYTO-014 noise gate: the summary `logr_dev` also measures cluster-file mismatch (GSE165845: 0.246, all samples `SampleQuality`) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. Pi tuning: mosaic prior / switch distance barely matter; CYTO-041 (noise-normalised BAF split, complete-loss promotion) cuts PASS mosaic calls by ~30 % without losing implanted ≥ 15 % mosaics. Two limits remain: (1) without phasing, < 20 % mosaic losses are not separable from regional BAF noise; (2) common CNVs (carrier frequency > 20 %) are almost never detected (1/236 DEL) because the median-LRR batch reference absorbs them; rare DEL (< 5 %) 38 %. | Open: rare-event sensitivity as primary metric; phasing-based mosaic BAF (`docs/validation/V2_cn.md`). |
 | K-026 | 2026-09-29 | Methylation QC (v2) | Spec deviations: vendor EPIC manifests put a 5th field on some control lines (≥ 4 fields accepted); detection p follows sesame's pOOBAH (1 − max(F(M), F(U))) rather than the ambiguous "1 − ECDF(M + U)"; minfi-style `Sentrix_ID`/`Sentrix_Position` sheet keys accepted. Against sesame 1.28.1 on minfiDataEPIC: β r = 0.99998 (max \|Δβ\| 0.011), detection calls 99.1 % identical, noob signals within 0.5 % (median) / 5 % (max). The systematic remainder is unexplained; it is not the negative controls (sesame did not attach the supplied control table). | Open; `docs/validation/V2_methylation.md`. Next step: dump sesame's background vectors and noob parameters to find the difference. |
@@ -44,8 +44,11 @@ Generated from `src/simurg/core/errors.py` — do not edit by hand.
 | `E_OUTPUT_EXISTS` | error | Outputs already exist; use --force or --resume | CLI-010 |
 | `E_PGX_BATCH_TOO_SMALL` | error | Too few (passing) samples in the batch for PGx CN calling | PCN-004 |
 | `E_PGX_CN_FAILED` | error | PGx CN calling failed for the sample | PCN-002 |
+| `E_PGX_CN_MISSING` | error | A sample has no PGx CN VCF (give --no-cn to run without) | PGX-011 |
 | `E_PGX_SOURCE` | error | PGx source tables missing; run scripts/pgx/fetch_sources.sh | PGX-001 |
 | `E_PGX_TRAIN_TOO_SMALL` | error | Too few (passing) samples to train a PGx CN model | PCN-005 |
+| `E_PGX_VCF_UNINDEXED` | error | SNV VCFs for PGx must be bgzipped and indexed | PGX-011 |
+| `E_PGX_VCF_UNSUITABLE` | error | SNV VCF unsuitable for PGx (squashing/locus filters/foreign) | PGX-011 |
 | `E_QC_CONFIG_EXT` | error | QC config file must be .yaml or .yml | QC-021 |
 | `E_QC_CONFIG_MISSING` | error | QC config file does not exist | QC-021 |
 | `E_QC_CONFIG_PARSE` | error | QC config file is not valid YAML | QC-021 |
@@ -0,0 +1,58 @@
+# V2 — PGx DB and star-allele calling (PGX-001 … PGX-021)
+
+Status: **implemented for CPIC star-allele genes, interim validation** (2026-09-29).
+
+* `scripts/pgx/fetch_sources.sh` → CPIC API tables (CC0, data version 2026-08-03).
+* `simurg pgx db build` → deterministic SQLite: 22 genes, 1354 alleles, 1897 normalised variant rows; GSA v4 PGx
+  probe map 5002 links; two builds byte-identical.
+* `simurg pgx star-allele call` → `diplotype_summary.tsv`, `pgx_calls.json` (11 star-allele genes).
+* Unit tests: `tests/unit/test_pgx_db.py`, `tests/unit/test_pgx_star.py`.
+
+## Data and truth
+
+* Arrays: the GSA v4 PGx demo (36 arrays = 30 CEPH/1000G individuals, `scripts/pgx/demo36_identity.tsv`), SNV VCFs
+  and PGx CN VCFs from SimurgArray.
+* Star-allele truth: GeT-RM consolidated PGx/HLA table (CDC, 2025 V4) — 16 of the 30 individuals
+  (`scripts/pgx/getrm_truth.py`).
+* Probe mask (PGX-010): array vs 1000G 30x (NYGC phased panel, fetched per gene over HTTPS) for the **other 14
+  individuals** (`scripts/pgx/derive_masks.py`), so the mask is independent of the GeT-RM validation set: 8 of 658
+  assessed records (23 probe names) below 95 % concordance — e.g. CYP2C19 rs367543002 and CYP2D6 rs1135836 called
+  homozygous ALT in every sample, the UGT1A1 `C>CATAT` TA-repeat probe. Regenerate with the command in the script
+  header (the mask lists vendor probe names and is not committed).
+
+## Result (16 individuals; replicate arrays counted once)
+
+| Gene | Top solution = GeT-RM | Truth among array-equivalent feasible solutions |
+|---|---|---|
+| CYP2C9 | **15/15** | 15/15 |
+| CYP2C19 | **14/15** | 15/15 |
+| CYP2D6 | **12/15** | 14/15 |
+| CYP3A5 | 15/15 | 15/15 |
+| TPMT | 15/16 | 15/16 |
+| CYP2B6 | 12/14 | 12/14 |
+| CYP4F2 | 8/9 | 8/9 |
+| NAT2 | 8/9 | 8/9 |
+| NUDT15 | 1/1 | 1/1 |
+| **total** | **100/109 (91.7 %)** | **103/109 (94.5 %)** |
+
+PGX-032 target (≥ 90 % for CYP2D6/CYP2C19/CYP2C9 against truth): **41/45 = 91 %** (top solution).
+Details and every discordant call: `V2_pgx_star_getrm16.json`.
+
+Not in the total (GeT-RM uses pre-2023 names the CPIC definitions no longer carry): UGT1A1 (GeT-RM `*60`;
+0/9 top) and SLCO1B1 (`*1B/*17/*21` vs CPIC `*37/*15/*20`; 9/15 top).
+
+Remaining discordances: CYP2D6 `*3/(*68)+*4` (NA12878; *68 hybrids not modelled, *3 frameshift probe), CYP2D6 *2 vs
+*1 where the array cannot separate them (prior picks *1), CYP2C19 *8, TPMT *21, CYP2B6 *27, CYP4F2 *2 — rare
+alleles whose defining variants are absent or masked on this array.
+
+## Error model (K-029)
+
+After masking, 0 of 9 740 array calls at DB variants disagreed with 1000G 30x in any GenCall-score bin (0.15–1.0),
+so the per-call error is 1e-3 (GS ≥ 0.15) and errors are handled by the probe mask. The first model,
+ε = 0.5·(1 − GS)², let correct low-GS calls lose to the prior (CYP2C9 7/15 → 15/15 after the change).
+
+## Caveats
+
+* 16 individuals, one array type, European (CEPH) only; masks from 14 individuals (spec: ≥ 100 samples).
+* Not yet: named-haplotype (G6PD), variant-based and tagging genes; CYP2D6 hybrids (*13, *36, *68); PharmVar-only
+  genes (CYP2A6, CYP3A4, CYP1A2, …); phenotype annotation (PGX-025) and the JSON schema (PGX-028).
@@ -0,0 +1,240 @@
+{
+ "individuals": 30,
+ "genes": {
+  "CYP2B6": {
+   "n": 14,
+   "top": 12,
+   "equivalent": 12,
+   "not_comparable": 1
+  },
+  "CYP2C19": {
+   "n": 15,
+   "top": 14,
+   "equivalent": 15
+  },
+  "CYP2C9": {
+   "n": 15,
+   "top": 15,
+   "equivalent": 15
+  },
+  "CYP2D6": {
+   "n": 15,
+   "top": 12,
+   "equivalent": 14
+  },
+  "CYP3A5": {
+   "n": 15,
+   "top": 15,
+   "equivalent": 15
+  },
+  "CYP4F2": {
+   "n": 9,
+   "top": 8,
+   "equivalent": 8
+  },
+  "NAT2": {
+   "n": 9,
+   "top": 8,
+   "equivalent": 8
+  },
+  "NUDT15": {
+   "n": 1,
+   "top": 1,
+   "equivalent": 1
+  },
+  "SLCO1B1": {
+   "n": 15,
+   "top": 9,
+   "equivalent": 9
+  },
+  "TPMT": {
+   "n": 16,
+   "top": 15,
+   "equivalent": 15
+  },
+  "UGT1A1": {
+   "n": 9,
+   "top": 0,
+   "equivalent": 1
+  }
+ },
+ "legacy_nomenclature_genes": [
+  "SLCO1B1",
+  "UGT1A1"
+ ],
+ "total": {
+  "n": 109,
+  "top": 100,
+  "equivalent": 103
+ },
+ "discordant": [
+  {
+   "individual": "NA12156",
+   "gene": "CYP4F2",
+   "truth": "(*2)/*3",
+   "call": "*1/*3",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12156",
+   "gene": "SLCO1B1",
+   "truth": "*1/*21",
+   "call": "*1/*37",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12156",
+   "gene": "UGT1A1",
+   "truth": "*60/*28",
+   "call": "*28/*80",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12044",
+   "gene": "TPMT",
+   "truth": "*1/*21",
+   "call": "*1/*1",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA07019",
+   "gene": "UGT1A1",
+   "truth": "*1/*60",
+   "call": "*1/*1",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA06991",
+   "gene": "UGT1A1",
+   "truth": "*60/*60",
+   "call": "*1/*1",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12878",
+   "gene": "CYP2D6",
+   "truth": "*3/(*68)+*4",
+   "call": "*4x2/*1;*4x2/*82",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12878",
+   "gene": "UGT1A1",
+   "truth": "*60/*28",
+   "call": "*28/*80",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA11993",
+   "gene": "SLCO1B1",
+   "truth": "*1/*17",
+   "call": "*1/*15",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA11993",
+   "gene": "UGT1A1",
+   "truth": "(*28 + *60)/*60",
+   "call": "*28/*80",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12003",
+   "gene": "UGT1A1",
+   "truth": "*60/*28",
+   "call": "*28/*80",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA07029",
+   "gene": "CYP2B6",
+   "truth": "*6/(*27)",
+   "call": "*1/*6",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA07029",
+   "gene": "CYP2C19",
+   "truth": "*8/*17",
+   "call": "*1/*17",
+   "within_equivalents": true
+  },
+  {
+   "individual": "NA07029",
+   "gene": "UGT1A1",
+   "truth": "*1/*1",
+   "call": "*1/*28",
+   "within_equivalents": true
+  },
+  {
+   "individual": "NA12892",
+   "gene": "CYP2B6",
+   "truth": "*1/*1",
+   "call": "*1/*5",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA11832",
+   "gene": "NAT2",
+   "truth": "*5/(*12)",
+   "call": "*1/*5",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA11832",
+   "gene": "SLCO1B1",
+   "truth": "*1/*1",
+   "call": "*14/*20",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA11832",
+   "gene": "UGT1A1",
+   "truth": "(*28 + *60)/*60",
+   "call": "*28/*80",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA10838",
+   "gene": "CYP2D6",
+   "truth": "*2/*4",
+   "call": "*1/*4;*2/*4",
+   "within_equivalents": true
+  },
+  {
+   "individual": "NA10838",
+   "gene": "SLCO1B1",
+   "truth": "*1/*1",
+   "call": "*1/*14",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12813",
+   "gene": "CYP2D6",
+   "truth": "*2/*4",
+   "call": "*1/*4;*2/*4",
+   "within_equivalents": true
+  },
+  {
+   "individual": "NA12813",
+   "gene": "SLCO1B1",
+   "truth": "*1/*21",
+   "call": "*1/*20",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA12813",
+   "gene": "UGT1A1",
+   "truth": "(*28 + *60)/(*28 + *60)",
+   "call": "*80/*80",
+   "within_equivalents": false
+  },
+  {
+   "individual": "NA11881",
+   "gene": "SLCO1B1",
+   "truth": "*1/(*15)",
+   "call": "*14/*15",
+   "within_equivalents": false
+  }
+ ]
+}