SimurgArray/docs/known_unknowns.md

Known unknowns and findings log

Plan §20 lists the open questions; this file records what the implementation has learned. Newest first.

ID	Date	Topic	Finding	Status
K-029	2026-09-29	PGx star alleles (v2)	(1) PGX-012 error model: ε(GS) = 0.5·(1 − GS)² was replaced by ε = 1e-3 (GS ≥ 0.15) / 0.05 — after probe masking 0 of 9 740 GSA v4 PGx calls disagreed with 1000G 30x at any GS, and the old model made correct low-GS calls lose to the prior (spec amended). (2) PGX-010 masks derived from 14 individuals (spec: ≥ 100); a variant absent from the filtered 1000G panel counts as hom-ref only if rare (CYP2B6 rs2279343 is filtered out, not absent). (3) CPIC core-allele definitions use IUPAC wildcards; alleles without definitions (NUDT15 *2, structural CYP2D6) are not callable from SNVs. (4) GeT-RM uses legacy UGT1A1/SLCO1B1 names. Result vs GeT-RM (16 CEPH): 100/109 gene calls, CYP2D6/2C19/2C9 41/45. (5) DPWG: no public machine-readable genotype→phenotype mapping; CPIC mappings are used for genes CPIC covers (as ClinPGx does), DPWG-only genes (CYP1A2, CYP3A4, F5) get no phenotype. (6) GeT-RM definitions are older than CPIC/PharmVar for UGT1A1, SLCO1B1, CYP1A2 and CYP2C8 (e.g. PharmVar CYP2C8 *19 = rs2071426 alone). Result vs GeT-RM (16 CEPH, 21 genes): 154/168 comparable gene calls. Independent check on GSE165845 (GSA v1 C2, 120 individuals × 3, WGS-derived truth, 2-fold masks): 95.7 % array-resolvable concordance, 67.9 % top-solution (the chip lacks many defining variants), replicate concordance 99.2 % without the failed array.	Open: KNMP DPWG tables if they become available (docs/validation/V2_pgx_star.md).
K-028	2026-09-29	PGx CN (v2)	Deviations from PCN-009: the per-probe reference is the training cohort's median stored in the model (a calling-batch median is not CN 2 where deletions are common, e.g. GSTM1 null ≈ 50 %); no GC-correction step; the CN mixture is ours: µ_c = δ + γ·log2((c+1)/3) per region with a free lower CN-0 component and Hardy–Weinberg-constrained weights (probe compression differs by region: γ ≈ 0.19 CYP2D6.p5 … 0.56 GSTT1). Interim validation on the GSA v4 PGx demo — whose 36 arrays turned out to be 30 CEPH/1000G individuals — against NYGC 30x SV RD_CN: 224/240 region calls (GSTM1, UGT2B17 30/30), but in-sample (trained on the same arrays, < 96 samples, priors chosen with this truth in view).	Open: validate on an independent GSA-PGx cohort; CYP2D6 via GeT-RM star alleles (docs/validation/V2_pgx_cn.md).
K-027	2026-09-29	CN/LOH (v2)	CYTO-014 noise gate: the summary logr_dev also measures cluster-file mismatch (GSE165845: 0.246, all samples SampleQuality) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. Pi tuning: mosaic prior / switch distance barely matter; CYTO-041 (noise-normalised BAF split, complete-loss promotion) cuts PASS mosaic calls by ~30 % without losing implanted ≥ 15 % mosaics. Two limits remain: (1) without phasing, < 20 % mosaic losses are not separable from regional BAF noise; (2) common CNVs (carrier frequency > 20 %) are almost never detected (1/236 DEL) because the median-LRR batch reference absorbs them; rare DEL (< 5 %) 38 %.	Open: rare-event sensitivity as primary metric; phasing-based mosaic BAF (docs/validation/V2_cn.md).
K-026	2026-09-29	Methylation QC (v2)	Spec deviations: vendor EPIC manifests put a 5th field on some control lines (≥ 4 fields accepted); detection p follows sesame's pOOBAH (1 − max(F(M), F(U))) rather than the ambiguous "1 − ECDF(M + U)"; minfi-style Sentrix_ID/Sentrix_Position sheet keys accepted. Against sesame 1.28.1 on minfiDataEPIC: β r = 0.99998 (max |Δβ| 0.011), detection calls 99.1 % identical, noob signals within 0.5 % (median) / 5 % (max). The systematic remainder is unexplained; it is not the negative controls (sesame did not attach the supplied control table).	Open; docs/validation/V2_methylation.md. Next step: dump sesame's background vectors and noob parameters to find the difference.
K-025	2026-09-28	Scaling limits (P9)	With the Rust kernels a Graviton3 core genotypes 78–80 GSA samples/min. At > 8 workers a 125 MB/s gp3 volume caps the run at ~620 samples/min (GTC ≈ 13 MB each); on a RAM disk efficiency is 0.77 at 16 and 0.48 at 32 workers for 360 samples, explained by a fixed start-up cost (≈ 2.5 s + 0.2 s per worker). Linear scaling for large batches is inferred from that model, not measured.	docs/validation/P9.md; measure a ≥ 5000-sample batch when one is at hand.
K-024	2026-09-28	GSE165845 (P8)	GSA-MD v1 chips have no public cluster file; the GSA v1 C1 EGT with the C2 build-38 BPM covers 99.996 % of bead addresses and gives median LogR deviation 0.246 (cluster mismatch, not noise). GSM5051309 (NA20362 rep 2) is a failed array (call rate 0.762, concordance 0.336 with its own replicates). 27 510 array records have no NYGC 30x record and are scored as hom-ref; 1444 of them are systematically called alt.	Recorded in docs/validation/P8.md; results reported with and without these.
K-023	2026-09-28	P6 optional outputs	Implemented and validated the same day: --info-fields (with --egt), --extra-format, --merge-snv-indel, --auxiliary-loci, PLINK 2 .pgen. Two decisions: OUT-035 rule (d) uses the MNV's last plus-strand base for minus-strand designs (spec amended); --extra-format writes NORMX/NORMY missing where the raw intensities are both 0, and THETA/R missing where both normalized intensities are 0 (148 values on 3 GSA samples; gtc2vcf writes 0).	resolved
K-022	2026-09-28	VCF oracles' limits (GSA v4 PGx, GRCh38)	gtc2vcf leaves 144 indel probes unresolved (REF=I/D) that SimurgArray resolves from SourceSeq; both leave 49 indels unresolved. GTCtoVCF (python-2.7 era) runs under Python 3 only with two local patches (pyvcf3 Format field; skip instead of crash on 3 CYP2D6 indels whose SourceSeq has the dash on the right, [ALLELE/-], which SimurgArray handles), and needs the alt-contig loci filtered when the FASTA has no alts. Neither oracle applies ploidy (males' X/Y/MT hets stay diploid).	documented
K-021	2026-09-28	Control metrics on legacy Omni2.5 IDATs	(1) The Omni2.5 manifest lists two bead addresses per control (e.g. 27630314 and 127630314, two bead-pool versions); only one is on a given chip. Absent addresses were NaN and poisoned the bead mean → 1794/2141 samples falsely flagged on both staining metrics. Absent addresses are now skipped. (2) 1290 samples have Biotin (Bgnd) green = 0 exactly (background-subtracted), so staining green is signal/0; QC-008 now defines that as +Inf (pass) and only 0/0 as NaN (flag).	resolved (spec QC-008 amended)
K-020	2026-09-28	QC report scope (P5b)	Implemented: views A–E, linked selection, thresholds modal (apply/download/upload YAML), chip/plate heatmaps, trend table and Tukey box plots, PCA card, xlsx/csv table, privacy options, --split-by dataset. Not yet: client-side xlsx export (CSV only), density mode for > 12 000 points (WebGL scatter is used), `--split-by plate	run`, X/Y median R for the sex card (the summary does not carry them; the card shows the sex check instead), methylation datasets (v2).
K-019	2026-09-28	GSA demo relatedness	The 36-sample GSA v4 PGx demo set contains duplicate/replicate pairs (KING φ ≈ 0.50, concordance ≥ 0.9999) and first-degree relatives (φ ≈ 0.25); without a pedigree they are flagged duplicate_of:/unexpected_relatedness as designed. Useful as a small relatedness fixture.	documented
K-018	2026-09-28	Locus-summary ChiTest_P100	Against BeadArrayFiles locus_summary.py all columns are bit-identical on 36 GSA samples (683 054 loci) except ChiTest_P100, which differs by ≤ 3.6e-15 relative in ~15 % of loci: scipy 1.18's ndtr matches neither libm nor a cephes erf/erfc port bit-for-bit. Accepted (p-value).	accepted
K-017	2026-09-28	BAF-regression median quirk	gtc2vcf BAFregress (1) compares AF with the threshold as a float (0.1f), and (2) for an even number of AA/BB calls averages the quickselect result with whatever element klib ks_ksmall leaves just below it, not the lower middle value. Both reproduced (--median reference, default): 36/36 GSA samples identical to the plugin at its printed precision; --median exact gives the true median. SimurgArray uses autosomal sites only in pipelines; the plugin uses every VCF record.	resolved
K-016	2026-09-28	Sex rule of AutoCall 1.6.2.2	Rule 1 (X heterozygosity) matches all 2141 vendor GTCs; rule 2 (X/Y intensity) differs in 12.	resolved
K-015	2026-09-28	Legacy GTC v3 statistics	AutoCall 1.6.2.2 writes GC10 = GC50 = 0 and #calls = #no-calls = 0 in GTC v3 (it does fill LogR dev and #intensity-only). gtc2vcf's AutoConvert 1.6.3.1 preset fills them, so this is specific to 1.6.2.x. Emulated by --preset autocall-1.6.2; the sample summary still reports the real statistics.	resolved (2138/2141 byte-identical with the preset, 2026-09-28)
K-014	2026-09-28	NaN sign	NaNs produced by libm (e.g. log2 of a negative extrapolated R) take the platform's default sign: + on aarch64, − on x86-64. SimurgArray always writes 0xFFC00000 (the reference's -NAN, x86 behaviour). The aarch64-built oracle differs in 1 locus of 9 GSA samples; oracle tests compare with NaNs canonicalised.	resolved
K-013	2026-09-27	Normalization version of public vendor GTCs	The 1000 Genomes Omni2.5 GTCs (AutoCall 1.6.2.2) are reproduced bit-for-bit by normalization 1.1.2 (109 191/109 191 bins, reserved fields included).	resolved
K-012	2026-09-27	FMA and bit-exactness	gcc on aarch64 contracts a*b+c into FMA by default; the gtc2vcf oracle then differs from the vendor by up to 4 ULP in some bins. Oracles are built with -ffp-contract=off. Any future C/Rust port (P9) must also forbid FMA contraction in exact mode.	resolved; constraint for P9
K-011	2026-09-27	Plan U1 (channel order)	Infinium II X = Red, Y = Grn confirmed on 2141 vendor GTCs (the GTC PDF's "Raw X = green" wording is wrong).	resolved
K-010	2026-09-27	Ingest throughput	On AWS, ingest of 2141 Omni samples was I/O-bound (gp3 125 MB/s baseline): 18.9 min, 4 of 16 cores busy. For P9 benchmarks provision gp3 throughput or use instance-store NVMe.	open (P9)
K-009	2026-09-27	Missing-intensity rule	3 loci (in 3 of 2141 vendor GTCs, AutoCall 1.6.2.2) keep one channel's intensity where we zero both. Hypothesis: old AutoCall applies the NBeads ≥ 2 rule per channel. Confirm with the bead counts of 5767788121_R01C01 (rs178512), 5826354152_R02C01 (SNP12-29082770), 6268201099_R02C01 (rs17133206); if confirmed, add a --missing-rule per-channel compatibility option.	open
K-008	2026-09-27	Vendor GTC layout	2141 vendor GTC v3 files (AutoCall 1.6.2.2) re-encode byte-identically with the canonical v3 order, so the writer reproduces the vendor layout (no padding or reordering).	resolved
K-007	2026-09-27	Demo data	The "GDA v1 Demo 36" download is a GenomeStudio project (.bsc + Data/*.bin), not IDATs. The GSA v4 PGx demo (idats_Demo_36.zip) is used as the P0 IDAT fixture instead. Bioconductor hapmap370k is no longer in the current release.	resolved (plan §17 updated)
K-006	2026-09-27	Sample sheet	Recognised input columns (Sample_ID, Sample_Name, …) collide case-insensitively with output columns sample_id/sample_name (IN-005). Recognised columns are exempt from the reserved-name check and are passed through verbatim.	decision, revisit in P4 (OUT-060 pass-through)
K-005	2026-09-27	Sample sheet	Illumina's GSA v4 template sheet has an empty Sample_Name column → E_SHEET_NAME_EMPTY (IN-010 behaviour as specified).	as specified
K-004	2026-09-27	GTC scanner data	PMT gains are 4 bytes read as int32 by BeadArrayFiles and written as float by gtc2vcf. We keep the raw 4 bytes (byte-exact round-trip) and expose the int32 view.	resolved
K-003	2026-09-27	Assay type from sequences	gtc2vcf's rule for inferring Infinium I colour from probe/source sequences (needed only for BPM locus entries < v6) agrees with the explicit BPM value for 99.71 % of GSA v4 Infinium I loci. The BPM is authoritative whenever it carries the assay type.	documented; test threshold 99.5 %
K-002	2026-09-27	EGT data block (plan U3)	GSA v4 and GDA v1 EGTs are data-block v9 with 15 extra floats per record; the first extra float is the intensity threshold (non-zero for 16 015 GDA loci). The reader tries candidate record sizes and accepts the one whose trailing cluster counts match. The Omni2.5 EGT is dbv 7 with 15 extra floats (structure confirmed); whether its first extra float is an intensity threshold (gtc2vcf) or unused (BeadArrayFiles) is still open.	partly resolved
K-001	2026-09-27	IDAT MidBlock (plan U2)	In GSA v4 (BeadChip 48x4), HumanCNV370 (2x12) and 450k (12x8) IDATs the MidBlock count equals N and the section is 4 bytes per entry (int32, as in illuminaio). gtc2vcf reads uint8 but never uses the values. We infer the element size from the section length.	resolved for these chips + BeadChip 4x10 (Omni2.5, 2141 files)