kemal
85229fba84
Raw X/Y and controls assembled from IDATs equal the vendor GTC arrays for 2138/2141 samples x 2.45M loci; 3 single-locus differences are recorded as K-009 (probable per-channel missing rule in AutoCall 1.6.2.2). Channel order X=Red confirmed (U1). `simurg ingest` processed all 2141 samples in 19 min (I/O-bound). AWS run cost ~$0.34 and left no resources.
2.3 KiB
2.3 KiB
Phase P2 validation report — canonical model and raw X/Y assembly
Date: 2026-09-27. Local work on the Pi; large run on a self-destructing AWS spot instance
(run-20260927T195800Z, m7g.4xlarge, 77 min, ≈ $0.34). Summary: P2_omni2141_summary.json.
Delivered
| Item | Module |
|---|---|
| Raw X/Y + control assembly (plan §7.1), per-chip address-map cache | signal/raw_xy.py |
| IDAT ↔ manifest address-coverage gate (IN-027) | signal/raw_xy.py |
| Canonical Arrow schemas, deterministic Parquet (plan §5) | core/schemas.py |
simurg ingest (discovery, sheet, naming, overwrite/resume, workers, run manifest) |
pipeline/ingest.py, cli/main.py |
Acceptance (plan §18 P2: raw X/Y exactly equal to vendor GTC TOC 1000/1001, 100 % loci × ≥ 50 samples)
| Check | Data | Result |
|---|---|---|
| Raw X/Y and controls vs oracle GTCs | 18 GSA v4 samples (gtc2vcf GTCs) | EXACT, 18/18 |
| Raw X/Y vs vendor GTCs (AutoCall 1.6.2.2) | 2141 Omni2.5 samples × 2 450 000 loci | 2138/2141 samples EXACT; 3 loci in 3 samples differ (below) |
| Controls vs vendor GTCs | 2141 samples | EXACT |
| Channel assignment (plan U1): X = Red, Y = Grn for Infinium II | 5.2 × 10⁹ locus values | confirmed (0 swapped loci) |
ingest outputs identical across thread counts and layouts |
synthetic + GSA | byte-identical |
The gate "100 % loci × ≥ 50 samples" is met (2138 samples fully exact). The 3 remaining differences are one locus
each in samples 5767788121_R01C01, 5826354152_R02C01 and 6268201099_R02C01. In each, we mark the locus missing
(0, 0) but the vendor kept one channel (e.g. rs178512: vendor [7086, 0]). Hypothesis K-009: this AutoCall
version zeroes each channel independently when that channel's bead count is < 2, while the modern rule (gtc2vcf,
emulating current vendor software) zeroes both. Not yet confirmed (needs the bead counts of those three IDATs).
simurg ingest at scale (2141 Omni2.5 samples, 16 workers)
| Metric | Value |
|---|---|
| Wall time | 18 min 52 s (0.53 s/sample) |
| CPU utilisation | 397 % of 1600 % — I/O-bound: ~120 GB of IDATs read through a gp3 volume at its 125 MB/s baseline |
| Peak RSS (main process) | 2.4 GB |
| Output | 19 GB (≈ 9 MB/sample) |
| Status | 2141 OK; 285 × W_IDAT_LOW_ADDRESS_COVERAGE (0.03 % of addresses absent), 4 × W_IDAT_NO_SCAN_TIME |