kemal/SimurgArray
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

P1: file parsers, sample sheet, discovery, CLI inspect
ci / test (3.12) (push) Failing after 11s
Details
ci / test (3.13) (push) Failing after 12s
Details

Browse Source 
Readers for IDAT v3, BPM (locus entries v4-v8), CSV manifest, EGT
(data blocks 5-9) and GTC v3-v5 (plus a layout-preserving GTC writer);
sample-sheet validation (IN-001..IN-014), IDAT discovery and pairing
(IN-020..IN-022), message catalogue and exit codes (CLI-014/016),
JSON Lines logging (CLI-007/008), atomic writes (CLI-011), run manifest,
and `simurg inspect`.

Validation (docs/validation/P1.md): GTC, BPM and EGT readers agree
exactly with Illumina BeadArrayFiles on GSA v4 / GDA v1 files; GTCs
round-trip byte-identically; BPM and CSV manifests are consistent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
This commit is contained in:
kemalandClaude Opus 5.5 committed 2026-09-27 16:56:39 +01:00
1 parent f10c651f0f
commit 4357d3deef
40 files changed
+4742
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/known_unknowns.md
+13
View File
@@ -0,0 +1,13 @@
# Known unknowns and findings log
Plan §20 lists the open questions; this file records what the implementation has learned. Newest first.
| ID | Date | Topic | Finding | Status |
|---|---|---|---|---|
| K-007 | 2026-09-27 | Demo data | The "GDA v1 Demo 36" download is a GenomeStudio project (`.bsc` + `Data/*.bin`), not IDATs. The GSA v4 PGx demo (`idats_Demo_36.zip`) is used as the P0 IDAT fixture instead. Bioconductor `hapmap370k` is no longer in the current release. | resolved (plan §17 updated) |
| K-006 | 2026-09-27 | Sample sheet | Recognised input columns (`Sample_ID`, `Sample_Name`, …) collide case-insensitively with output columns `sample_id`/`sample_name` (IN-005). Recognised columns are exempt from the reserved-name check and are passed through verbatim. | decision, revisit in P4 (OUT-060 pass-through) |
| K-005 | 2026-09-27 | Sample sheet | Illumina's GSA v4 template sheet has an empty `Sample_Name` column → `E_SHEET_NAME_EMPTY` (IN-010 behaviour as specified). | as specified |
| K-004 | 2026-09-27 | GTC scanner data | PMT gains are 4 bytes read as int32 by BeadArrayFiles and written as float by gtc2vcf. We keep the raw 4 bytes (byte-exact round-trip) and expose the int32 view. | resolved |
| K-003 | 2026-09-27 | Assay type from sequences | gtc2vcf's rule for inferring Infinium I colour from probe/source sequences (needed only for BPM locus entries < v6) agrees with the explicit BPM value for 99.71 % of GSA v4 Infinium I loci. The BPM is authoritative whenever it carries the assay type. | documented; test threshold 99.5 % |
| K-002 | 2026-09-27 | EGT data block (plan U3) | GSA v4 and GDA v1 EGTs are data-block v9 with 15 extra floats per record; the first extra float is the intensity threshold (non-zero for 16 015 GDA loci). The reader tries candidate record sizes and accepts the one whose trailing cluster counts match. dbv 7/8 files still untested. | partly resolved |
| K-001 | 2026-09-27 | IDAT MidBlock (plan U2) | In GSA v4 (BeadChip 48x4), HumanCNV370 (2x12) and 450k (12x8) IDATs the MidBlock count equals N and the section is 4 bytes per entry (int32, as in illuminaio). gtc2vcf reads uint8 but never uses the values. We infer the element size from the section length. | resolved for these chips |
docs/messages.md
+47
View File
@@ -0,0 +1,47 @@
# SimurgArray message catalogue
Generated from `src/simurg/core/errors.py` — do not edit by hand.
| Code | Level | Meaning | Spec |
|---|---|---|---|
| `E_BPM_CSV_MISMATCH` | error | BPM and CSV manifests describe different loci | IN-042 |
| `E_BPM_FORMAT` | error | BPM manifest is malformed or unsupported | IN-040 |
| `E_CONFIG_KEY` | error | Unknown key in --config YAML | CLI-019 |
| `E_CONFIG_PARSE` | error | --config file is not valid YAML | CLI-019 |
| `E_CSV_MANIFEST_FORMAT` | error | CSV manifest is malformed | IN-041 |
| `E_DUP_GTC` | error | Several GTC files resolve to the same sample | IN-030 |
| `E_DUP_IDAT` | error | Several IDAT pairs resolve to the same sample | IN-022 |
| `E_EGT_LOAD` | error | Cluster (EGT) file cannot be loaded | IN-044 |
| `E_FILE_NOT_FOUND` | error | Input file or folder does not exist | CLI-016 |
| `E_FILE_UNREADABLE` | error | Input file cannot be read | CLI-016 |
| `E_GTC_CORRUPT` | warning | GTC file failed validation; sample skipped | IN-032 |
| `E_GTC_UNSUITABLE` | error | GTC lacks fields required by this command | IN-031 |
| `E_IDAT_ENCRYPTED` | error | Encrypted (version 1) IDATs are not supported | IN-023 |
| `E_IDAT_FORMAT` | error | IDAT file is malformed | IN-023 |
| `E_IDAT_PAIR_MISMATCH` | error | Red and Grn IDATs of a sample do not match | IN-023 |
| `E_INTERNAL` | error | Internal error (bug); rerun with --debug and report | CLI-014 |
| `E_NO_INPUT` | error | Neither an input folder nor a sample sheet was given | CLI-006 |
| `E_NO_INPUT_FOLDER` | error | Sample sheet has no RootFolder and no folder was given | CLI-006 |
| `E_OUTPUT_EXISTS` | error | Outputs already exist; use --force or --resume | CLI-010 |
| `E_SHEET_BARCODE_FORMAT` | error | Invalid SentrixBarcode_A / SentrixPosition_A value | IN-007 |
| `E_SHEET_DELIMITER` | error | Sample sheet is not comma-delimited CSV | IN-001 |
| `E_SHEET_DUP_COLUMN` | error | Duplicate column (column names are case-insensitive) | IN-003 |
| `E_SHEET_DUP_SAMPLE` | error | The same barcode/position appears in several rows | IN-009 |
| `E_SHEET_EMPTY` | error | Sample sheet has no data rows | IN-001 |
| `E_SHEET_NAME_COLLISION` | error | Sample names are not unique or clash with sample IDs | IN-010 |
| `E_SHEET_NAME_EMPTY` | error | Sample_Name is empty in at least one row | IN-010 |
| `E_SHEET_NO_KEY_COLUMNS` | error | Sample sheet lacks SentrixBarcode_A/SentrixPosition_A | IN-002 |
| `E_SHEET_RESERVED` | error | Sample sheet column collides with an output column | IN-005 |
| `E_SHEET_TOO_LONG` | error | Sample sheet name or value longer than 500 characters | IN-006 |
| `I_UNLISTED_IDATS` | info | IDAT pairs not listed in the sample sheet are ignored | IN-016 |
| `W_DUP_IDAT_RESOLVED` | warning | Duplicate IDAT pair resolved by --on-duplicate-idat | IN-022 |
| `W_EGT_NAME` | warning | Cluster file name is not portable | IN-045 |
| `W_GTC_FOREIGN` | warning | GTC was produced by another tool | IN-031 |
| `W_IDAT_NAME_MISMATCH` | warning | IDAT barcode/position disagrees with its file name | IN-023 |
| `W_IDAT_NO_SCAN_TIME` | warning | IDAT has no scan entry in its run info | IN-024 |
| `W_IDAT_UNPAIRED` | warning | IDAT channel file without its partner | IN-021 |
| `W_MANIFEST_NO_REFSTRAND` | warning | Manifest lacks RefStrand annotation | IN-040 |
| `W_SHEET_BARCODE_REPAIRED` | warning | Barcode written as a decimal was repaired | IN-007 |
| `W_SHEET_EMPTY_COLUMN` | warning | Columns with empty headers are ignored | IN-004 |
| `W_SHEET_ROOTFOLDER_OVERRIDDEN` | warning | CLI folder overrides the sheet RootFolder | IN-008 |
| `W_SHEET_SEX_CONFLICT` | warning | Reported-sex columns disagree | IN-014 |
docs/validation/P1.md
+58
View File
@@ -0,0 +1,58 @@
# Phase P0/P1 validation report
Date: 2026-09-27 · Machine: Raspberry Pi 5 (aarch64, 4 cores, 8 GB) · Python 3.13, NumPy 2.2.
Commands: `pytest tests/unit` (every CI run), `pytest tests/oracle -m "not slow"` and `-m slow` (needs
`scripts/fetch_data.sh p0`, `scripts/bootstrap_refs.sh`, `scripts/make_oracle_gtcs.sh`).
## Scope delivered
| Plan §18 P1 item | Module | Status |
|---|---|---|
| Little-endian binary + .NET string codec | `formats/binutil.py` | done |
| IDAT v3 reader (gzip, metadata, RunInfo → ISO date, encrypted detection) | `formats/idat.py` | done |
| BPM reader (v3–5, locus entries v4/6/7/8, norm-bin rule incl. mod-256 wrap) | `formats/bpm.py` | done |
| CSV manifest reader + BPM↔CSV consistency (IN-042) | `formats/csv_manifest.py` | done |
| EGT reader (data-block variants, count cross-check) | `formats/egt.py` | done |
| GTC v3–v5 reader and writer (layout-preserving) | `formats/gtc.py` | done |
| Sample sheet (IN-001 … IN-014) | `formats/samplesheet.py` | done |
| IDAT discovery, pairing, duplicates, input-set rule (IN-020 … IN-022, CLI-006) | `formats/discovery.py` | done |
| Message catalogue, exit codes, JSON logging, rate limiting | `core/errors.py`, `core/log.py` | done |
| Atomic writes, overwrite policy, run manifest | `core/atomic.py`, `core/provenance.py` | done (used from P4 on) |
| `simurg version`, `simurg inspect [--json]` | `cli/main.py` | done |
| Synthetic IDAT/BPM/EGT writers for tests | `synth/files.py` | done |
## Acceptance results (plan §18 P1 gate)
| Criterion | Data | Result |
|---|---|---|
| GTC fields equal to BeadArrayFiles (23 fields incl. raw X/Y, genotypes, scores, BAF, LRR, base calls, sample stats, percentiles, 48 normalization transforms) | 18 GSA v4 GTCs from gtc2vcf `idat2gtc` (preset: Array Analysis CLI) | **EXACT, 18/18** |
| GTC read → write byte-identical | same 18 files; synthetic v3/v5 incl. non-canonical layouts; 25 random property cases | **EXACT** |
| BPM equal to BeadArrayFiles (names, SNP, Chr, MapInfo, AddressA, assay type, normalization lookups, RefStrand, manifest name, control config) | GSA v4 PGx BPM (683 054 loci) | **EXACT** |
| EGT equal to BeadArrayFiles (N, θ/R mean/SD ×3, intensity threshold, GenTrain score, cluster separation, address) | GSA v4 (683 054 records) and GDA v1 (1 904 599 records), ~5 000 evenly spaced records each | **EXACT** |
| BPM ↔ CSV consistency (IN-042: names, counts, Chr, MapInfo, SNP, AddressA/B) | GSA v4 | **0 differences** |
| All IDATs parse and pair; Red/Grn address sets equal; same address set on every chip | 72 GSA IDATs (36 samples, 2 chips) | **pass** |
| Manifest addresses present in IDATs | GSA v4 | **100 %** |
| Bioconductor fixtures: 381 079 / 622 399 addresses (plain + gzip); 2 encrypted files → `E_IDAT_ENCRYPTED` | IlluminaDataTestFiles 1.50.0 | **pass** |
| Malformed inputs never raise uncaught exceptions (truncation, bad magic, bad TOC offsets, bad string lengths, duplicates) | synthetic matrix (EDGE-002, EDGE-004) | **pass** |
| Sample-sheet matrix EDGE-007 (27 cases) incl. 10 000 rows < 2 s (PERF-007) | synthetic | **pass** |
| Locale independence of `inspect --json` (C, tr_TR, de_DE) | synthetic | **pass** |
## Performance on the Pi (single thread, pure Python/NumPy)
| Operation | Time | Peak RSS |
|---|---|---|
| IDAT read (734 656 addresses) | 3 ms | – |
| GTC read (683 054 loci) | 34 ms | – |
| EGT read GSA / GDA | 2.2 s / 6.1 s | – |
| CSV manifest read GSA | 12.5 s | 713 MB |
| BPM read GSA | 17.8 s | 665 MB |
BPM/CSV parsing is the only slow step; it is a one-off per run and moves to Rust in P9 (or is cached).
## Not yet covered (needs the larger machine / later phases)
- GEO sweep of 254 genotyping IDAT series and the 1000 Genomes Omni2.5 set (plan §18 P1 wording) — P1 is
validated on P0 data only; rerun `tests/oracle` after `scripts/fetch_data.sh p1`.
- GTCs written by vendor software (only gtc2vcf-written GTCs are available locally); the 1000 Genomes Broad set
contains vendor GTCs.
- IN-027 (IDAT ↔ manifest coverage gate) is P2; the coverage computation is already exercised by a test.