kemal
7bd73700b2
All 2141 vendor-written GTC v3 files match BeadArrayFiles field by field and re-encode byte-identically; all IDAT pairs parse; the 2.45M-locus BPM, dbv-7 EGT and CSV manifest agree with BeadArrayFiles and each other. Run on a self-destructing AWS spot instance (42 min, ~$0.20).
4.7 KiB
4.7 KiB
Phase P0/P1 validation report
Date: 2026-09-27 · Machine: Raspberry Pi 5 (aarch64, 4 cores, 8 GB) · Python 3.13, NumPy 2.2.
Commands: pytest tests/unit (every CI run), pytest tests/oracle -m "not slow" and -m slow (needs
scripts/fetch_data.sh p0, scripts/bootstrap_refs.sh, scripts/make_oracle_gtcs.sh).
Scope delivered
| Plan §18 P1 item | Module | Status |
|---|---|---|
| Little-endian binary + .NET string codec | formats/binutil.py |
done |
| IDAT v3 reader (gzip, metadata, RunInfo → ISO date, encrypted detection) | formats/idat.py |
done |
| BPM reader (v3–5, locus entries v4/6/7/8, norm-bin rule incl. mod-256 wrap) | formats/bpm.py |
done |
| CSV manifest reader + BPM↔CSV consistency (IN-042) | formats/csv_manifest.py |
done |
| EGT reader (data-block variants, count cross-check) | formats/egt.py |
done |
| GTC v3–v5 reader and writer (layout-preserving) | formats/gtc.py |
done |
| Sample sheet (IN-001 … IN-014) | formats/samplesheet.py |
done |
| IDAT discovery, pairing, duplicates, input-set rule (IN-020 … IN-022, CLI-006) | formats/discovery.py |
done |
| Message catalogue, exit codes, JSON logging, rate limiting | core/errors.py, core/log.py |
done |
| Atomic writes, overwrite policy, run manifest | core/atomic.py, core/provenance.py |
done (used from P4 on) |
simurg version, simurg inspect [--json] |
cli/main.py |
done |
| Synthetic IDAT/BPM/EGT writers for tests | synth/files.py |
done |
Acceptance results (plan §18 P1 gate)
| Criterion | Data | Result |
|---|---|---|
| GTC fields equal to BeadArrayFiles (23 fields incl. raw X/Y, genotypes, scores, BAF, LRR, base calls, sample stats, percentiles, 48 normalization transforms) | 18 GSA v4 GTCs from gtc2vcf idat2gtc (preset: Array Analysis CLI) |
EXACT, 18/18 |
| GTC read → write byte-identical | same 18 files; synthetic v3/v5 incl. non-canonical layouts; 25 random property cases | EXACT |
| BPM equal to BeadArrayFiles (names, SNP, Chr, MapInfo, AddressA, assay type, normalization lookups, RefStrand, manifest name, control config) | GSA v4 PGx BPM (683 054 loci) | EXACT |
| EGT equal to BeadArrayFiles (N, θ/R mean/SD ×3, intensity threshold, GenTrain score, cluster separation, address) | GSA v4 (683 054 records) and GDA v1 (1 904 599 records), ~5 000 evenly spaced records each | EXACT |
| BPM ↔ CSV consistency (IN-042: names, counts, Chr, MapInfo, SNP, AddressA/B) | GSA v4 | 0 differences |
| All IDATs parse and pair; Red/Grn address sets equal; same address set on every chip | 72 GSA IDATs (36 samples, 2 chips) | pass |
| Manifest addresses present in IDATs | GSA v4 | 100 % |
Bioconductor fixtures: 381 079 / 622 399 addresses (plain + gzip); 2 encrypted files → E_IDAT_ENCRYPTED |
IlluminaDataTestFiles 1.50.0 | pass |
| Malformed inputs never raise uncaught exceptions (truncation, bad magic, bad TOC offsets, bad string lengths, duplicates) | synthetic matrix (EDGE-002, EDGE-004) | pass |
| Sample-sheet matrix EDGE-007 (27 cases) incl. 10 000 rows < 2 s (PERF-007) | synthetic | pass |
Locale independence of inspect --json (C, tr_TR, de_DE) |
synthetic | pass |
Large-scale run: 1000 Genomes Omni2.5 (AWS, 2026-09-27)
Self-destructing spot instance (scripts/cloud/trial.sh, m7g.4xlarge, 16 vCPU), run run-20260927T183712Z,
summary in P1_omni2141_summary.json. 107 GB archive streamed from s3://1000genomes (extracted 170 GB) in
22 min; validation 18 min; instance lifetime 42 min; cost ≈ $0.20.
| Criterion | Result |
|---|---|
| 2141 vendor-written GTCs (v3, AutoCall 1.6.2.2): every field equal to BeadArrayFiles | 2141/2141 EXACT |
| Same GTCs: read → write byte-identical | 2141/2141 |
| 2141 IDAT pairs (BeadChip 4x10) parse and pair; MidBlock int32 | 2141/2141 |
| BPM HumanOmni2.5-4v1-Multi_B (2 450 000 loci, 51 normalization bins) vs BeadArrayFiles | EXACT |
| EGT (data block v7, 2 450 000 records) vs BeadArrayFiles: all θ/R means and SDs, N | 0 mismatches |
| BPM ↔ CSV consistency | 0 differences |
| GTC transforms per sample = BPM bins (51) | pass |
Performance on the Pi (single thread, pure Python/NumPy)
| Operation | Time | Peak RSS |
|---|---|---|
| IDAT read (734 656 addresses) | 3 ms | – |
| GTC read (683 054 loci) | 34 ms | – |
| EGT read GSA / GDA | 2.2 s / 6.1 s | – |
| CSV manifest read GSA | 12.5 s | 713 MB |
| BPM read GSA | 17.8 s | 665 MB |
BPM/CSV parsing is the only slow step; it is a one-off per run and moves to Rust in P9 (or is cached).
Not yet covered (needs the larger machine / later phases)
- GEO sweep of 254 genotyping IDAT series (other chip generations, IDAT variants) — still open.
- IN-027 (IDAT ↔ manifest coverage gate) is P2; the coverage computation is already exercised by a test.