kemal
2622dc5b88
P2 validation: oracle test and large-scale raw X/Y check
...
- tests/oracle/test_p2_raw_xy.py: IDAT-assembled raw X/Y and controls
equal the arrays in oracle GTCs
- validate_p1_large.py: phase-P2 comparison against vendor GTCs with
diagnostics (channel swap, bead-count disagreements)
- cloud trial: also runs `simurg ingest` on all samples (time/memory),
disk 250 GB
2026-09-27 20:57:42 +01:00
kemal
87f9b3499c
P2 (WIP): raw X/Y assembly, canonical Parquet model, simurg ingest
...
- signal/raw_xy.py: vectorised Infinium I/II raw X/Y and control assembly
(plan §7.1) with a per-chip address-map cache; IN-027 coverage gate
- core/schemas.py: Arrow schemas (loci, sample_intensity, controls),
deterministic atomic Parquet writing
- pipeline/ingest.py + CLI: discovery, sheet, IN-010 naming, overwrite
check before any work, spawn-based workers, per-sample isolation,
--resume by input hashes, run manifest, exit codes 0/3/4
- matches gtc2vcf raw X/Y and controls exactly on 18 GSA samples;
outputs byte-identical across thread counts and layouts
2026-09-27 20:54:06 +01:00
kemal
8cd092460f
P1: file parsers, sample sheet, discovery, CLI inspect
...
Readers for IDAT v3, BPM (locus entries v4-v8), CSV manifest, EGT
(data blocks 5-9) and GTC v3-v5 (plus a layout-preserving GTC writer);
sample-sheet validation (IN-001..IN-014), IDAT discovery and pairing
(IN-020..IN-022), message catalogue and exit codes (CLI-014/016),
JSON Lines logging (CLI-007/008), atomic writes (CLI-011), run manifest,
and `simurg inspect`.
Validation (docs/validation/P1.md): GTC, BPM and EGT readers agree
exactly with Illumina BeadArrayFiles on GSA v4 / GDA v1 files; GTCs
round-trip byte-identically; BPM and CSV manifests are consistent.
2026-09-27 16:56:39 +01:00
kemal