From fb1ff51b76715408229ff4c4f8831d096e28acd8 Mon Sep 17 00:00:00 2001 From: Kemal Yaylali Date: Sun, 27 Sep 2026 20:22:35 +0100 Subject: [PATCH] P1 validated on 1000 Genomes Omni2.5 (2141 vendor GTCs + IDATs) All 2141 vendor-written GTC v3 files match BeadArrayFiles field by field and re-encode byte-identically; all IDAT pairs parse; the 2.45M-locus BPM, dbv-7 EGT and CSV manifest agree with BeadArrayFiles and each other. Run on a self-destructing AWS spot instance (42 min, ~$0.20). --- docs/known_unknowns.md | 5 ++- docs/validation/P1.md | 21 ++++++++-- docs/validation/P1_omni2141_summary.json | 53 ++++++++++++++++++++++++ 3 files changed, 73 insertions(+), 6 deletions(-) create mode 100644 docs/validation/P1_omni2141_summary.json diff --git a/docs/known_unknowns.md b/docs/known_unknowns.md index ec4725d..47c55e5 100644 --- a/docs/known_unknowns.md +++ b/docs/known_unknowns.md @@ -4,10 +4,11 @@ Plan §20 lists the open questions; this file records what the implementation ha | ID | Date | Topic | Finding | Status | |---|---|---|---|---| +| K-008 | 2026-09-27 | Vendor GTC layout | 2141 vendor GTC v3 files (AutoCall 1.6.2.2) re-encode byte-identically with the canonical v3 order, so the writer reproduces the vendor layout (no padding or reordering). | resolved | | K-007 | 2026-09-27 | Demo data | The "GDA v1 Demo 36" download is a GenomeStudio project (`.bsc` + `Data/*.bin`), not IDATs. The GSA v4 PGx demo (`idats_Demo_36.zip`) is used as the P0 IDAT fixture instead. Bioconductor `hapmap370k` is no longer in the current release. | resolved (plan §17 updated) | | K-006 | 2026-09-27 | Sample sheet | Recognised input columns (`Sample_ID`, `Sample_Name`, …) collide case-insensitively with output columns `sample_id`/`sample_name` (IN-005). Recognised columns are exempt from the reserved-name check and are passed through verbatim. | decision, revisit in P4 (OUT-060 pass-through) | | K-005 | 2026-09-27 | Sample sheet | Illumina's GSA v4 template sheet has an empty `Sample_Name` column → `E_SHEET_NAME_EMPTY` (IN-010 behaviour as specified). | as specified | | K-004 | 2026-09-27 | GTC scanner data | PMT gains are 4 bytes read as int32 by BeadArrayFiles and written as float by gtc2vcf. We keep the raw 4 bytes (byte-exact round-trip) and expose the int32 view. | resolved | | K-003 | 2026-09-27 | Assay type from sequences | gtc2vcf's rule for inferring Infinium I colour from probe/source sequences (needed only for BPM locus entries < v6) agrees with the explicit BPM value for 99.71 % of GSA v4 Infinium I loci. The BPM is authoritative whenever it carries the assay type. | documented; test threshold 99.5 % | -| K-002 | 2026-09-27 | EGT data block (plan U3) | GSA v4 and GDA v1 EGTs are data-block v9 with 15 extra floats per record; the first extra float is the intensity threshold (non-zero for 16 015 GDA loci). The reader tries candidate record sizes and accepts the one whose trailing cluster counts match. dbv 7/8 files still untested. | partly resolved | -| K-001 | 2026-09-27 | IDAT MidBlock (plan U2) | In GSA v4 (BeadChip 48x4), HumanCNV370 (2x12) and 450k (12x8) IDATs the MidBlock count equals N and the section is 4 bytes per entry (int32, as in illuminaio). gtc2vcf reads uint8 but never uses the values. We infer the element size from the section length. | resolved for these chips | +| K-002 | 2026-09-27 | EGT data block (plan U3) | GSA v4 and GDA v1 EGTs are data-block v9 with 15 extra floats per record; the first extra float is the intensity threshold (non-zero for 16 015 GDA loci). The reader tries candidate record sizes and accepts the one whose trailing cluster counts match. The Omni2.5 EGT is dbv 7 with 15 extra floats (structure confirmed); whether its first extra float is an intensity threshold (gtc2vcf) or unused (BeadArrayFiles) is still open. | partly resolved | +| K-001 | 2026-09-27 | IDAT MidBlock (plan U2) | In GSA v4 (BeadChip 48x4), HumanCNV370 (2x12) and 450k (12x8) IDATs the MidBlock count equals N and the section is 4 bytes per entry (int32, as in illuminaio). gtc2vcf reads uint8 but never uses the values. We infer the element size from the section length. | resolved for these chips + BeadChip 4x10 (Omni2.5, 2141 files) | diff --git a/docs/validation/P1.md b/docs/validation/P1.md index 81198be..391f40c 100644 --- a/docs/validation/P1.md +++ b/docs/validation/P1.md @@ -37,6 +37,22 @@ Commands: `pytest tests/unit` (every CI run), `pytest tests/oracle -m "not slow" | Sample-sheet matrix EDGE-007 (27 cases) incl. 10 000 rows < 2 s (PERF-007) | synthetic | **pass** | | Locale independence of `inspect --json` (C, tr_TR, de_DE) | synthetic | **pass** | +## Large-scale run: 1000 Genomes Omni2.5 (AWS, 2026-09-27) + +Self-destructing spot instance (`scripts/cloud/trial.sh`, m7g.4xlarge, 16 vCPU), run `run-20260927T183712Z`, +summary in `P1_omni2141_summary.json`. 107 GB archive streamed from `s3://1000genomes` (extracted 170 GB) in +22 min; validation 18 min; instance lifetime 42 min; cost ≈ $0.20. + +| Criterion | Result | +|---|---| +| 2141 **vendor-written** GTCs (v3, AutoCall 1.6.2.2): every field equal to BeadArrayFiles | **2141/2141 EXACT** | +| Same GTCs: read → write byte-identical | **2141/2141** | +| 2141 IDAT pairs (BeadChip 4x10) parse and pair; MidBlock int32 | **2141/2141** | +| BPM HumanOmni2.5-4v1-Multi_B (2 450 000 loci, 51 normalization bins) vs BeadArrayFiles | **EXACT** | +| EGT (data block v7, 2 450 000 records) vs BeadArrayFiles: all θ/R means and SDs, N | **0 mismatches** | +| BPM ↔ CSV consistency | **0 differences** | +| GTC transforms per sample = BPM bins (51) | **pass** | + ## Performance on the Pi (single thread, pure Python/NumPy) | Operation | Time | Peak RSS | @@ -51,8 +67,5 @@ BPM/CSV parsing is the only slow step; it is a one-off per run and moves to Rust ## Not yet covered (needs the larger machine / later phases) -- GEO sweep of 254 genotyping IDAT series and the 1000 Genomes Omni2.5 set (plan §18 P1 wording) — P1 is - validated on P0 data only; rerun `tests/oracle` after `scripts/fetch_data.sh p1`. -- GTCs written by vendor software (only gtc2vcf-written GTCs are available locally); the 1000 Genomes Broad set - contains vendor GTCs. +- GEO sweep of 254 genotyping IDAT series (other chip generations, IDAT variants) — still open. - IN-027 (IDAT ↔ manifest coverage gate) is P2; the coverage computation is already exercised by a test. diff --git a/docs/validation/P1_omni2141_summary.json b/docs/validation/P1_omni2141_summary.json new file mode 100644 index 0000000..8f84747 --- /dev/null +++ b/docs/validation/P1_omni2141_summary.json @@ -0,0 +1,53 @@ +{ + "root": "/work/omni/data", + "started": "2026-09-27T19:00:42Z", + "n_gtc": 2141, + "n_idat_pairs": 2141, + "unpaired_idats": 0, + "gtc_ok": 2141, + "gtc_roundtrip_identical": 2141, + "gtc_versions": [ + 3 + ], + "gtc_autocall_versions": [ + "1.6.2.2" + ], + "gtc_failures": [], + "gtc_roundtrip_failures": [], + "idat_ok": 2141, + "idat_failures": [], + "idat_midblock_itemsizes": [ + 4 + ], + "idat_chip_types": [ + "BeadChip 4x10" + ], + "bpm": { + "num_loci": 2450000, + "names": true, + "snps": true, + "chroms": true, + "map_infos": true, + "address_a": true, + "assay_types": true, + "normalization_lookups": true, + "n_bins": 51 + }, + "egt": { + "num_records": 2450000, + "dbv": 7, + "extra_floats": 15, + "record_mismatches": 0 + }, + "bpm_csv_problems": [], + "gtc_vs_bpm_num_snps": [ + 2450000, + 2450000 + ], + "gtc_transforms_vs_bins": [ + 51, + 51 + ], + "seconds": 1098.7, + "passed": true +}