kemalandClaude Opus 5.5
5c93e194e0
Raw X/Y and controls assembled from IDATs equal the vendor GTC arrays for 2138/2141 samples x 2.45M loci; 3 single-locus differences are recorded as K-009 (probable per-channel missing rule in AutoCall 1.6.2.2). Channel order X=Red confirmed (U1). `simurg ingest` processed all 2141 samples in 19 min (I/O-bound). AWS run cost ~$0.34 and left no resources. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
18 lines
3.4 KiB
Markdown
18 lines
3.4 KiB
Markdown
# Known unknowns and findings log
|
|
|
|
Plan §20 lists the open questions; this file records what the implementation has learned. Newest first.
|
|
|
|
| ID | Date | Topic | Finding | Status |
|
|
|---|---|---|---|---|
|
|
| K-011 | 2026-09-27 | Plan U1 (channel order) | Infinium II X = Red, Y = Grn confirmed on 2141 vendor GTCs (the GTC PDF's "Raw X = green" wording is wrong). | resolved |
|
|
| K-010 | 2026-09-27 | Ingest throughput | On AWS, `ingest` of 2141 Omni samples was I/O-bound (gp3 125 MB/s baseline): 18.9 min, 4 of 16 cores busy. For P9 benchmarks provision gp3 throughput or use instance-store NVMe. | open (P9) |
|
|
| K-009 | 2026-09-27 | Missing-intensity rule | 3 loci (in 3 of 2141 vendor GTCs, AutoCall 1.6.2.2) keep one channel's intensity where we zero both. Hypothesis: old AutoCall applies the NBeads ≥ 2 rule per channel. Confirm with the bead counts of 5767788121_R01C01 (rs178512), 5826354152_R02C01 (SNP12-29082770), 6268201099_R02C01 (rs17133206); if confirmed, add a `--missing-rule per-channel` compatibility option. | open |
|
|
| K-008 | 2026-09-27 | Vendor GTC layout | 2141 vendor GTC v3 files (AutoCall 1.6.2.2) re-encode byte-identically with the canonical v3 order, so the writer reproduces the vendor layout (no padding or reordering). | resolved |
|
|
| K-007 | 2026-09-27 | Demo data | The "GDA v1 Demo 36" download is a GenomeStudio project (`.bsc` + `Data/*.bin`), not IDATs. The GSA v4 PGx demo (`idats_Demo_36.zip`) is used as the P0 IDAT fixture instead. Bioconductor `hapmap370k` is no longer in the current release. | resolved (plan §17 updated) |
|
|
| K-006 | 2026-09-27 | Sample sheet | Recognised input columns (`Sample_ID`, `Sample_Name`, …) collide case-insensitively with output columns `sample_id`/`sample_name` (IN-005). Recognised columns are exempt from the reserved-name check and are passed through verbatim. | decision, revisit in P4 (OUT-060 pass-through) |
|
|
| K-005 | 2026-09-27 | Sample sheet | Illumina's GSA v4 template sheet has an empty `Sample_Name` column → `E_SHEET_NAME_EMPTY` (IN-010 behaviour as specified). | as specified |
|
|
| K-004 | 2026-09-27 | GTC scanner data | PMT gains are 4 bytes read as int32 by BeadArrayFiles and written as float by gtc2vcf. We keep the raw 4 bytes (byte-exact round-trip) and expose the int32 view. | resolved |
|
|
| K-003 | 2026-09-27 | Assay type from sequences | gtc2vcf's rule for inferring Infinium I colour from probe/source sequences (needed only for BPM locus entries < v6) agrees with the explicit BPM value for 99.71 % of GSA v4 Infinium I loci. The BPM is authoritative whenever it carries the assay type. | documented; test threshold 99.5 % |
|
|
| K-002 | 2026-09-27 | EGT data block (plan U3) | GSA v4 and GDA v1 EGTs are data-block v9 with 15 extra floats per record; the first extra float is the intensity threshold (non-zero for 16 015 GDA loci). The reader tries candidate record sizes and accepts the one whose trailing cluster counts match. The Omni2.5 EGT is dbv 7 with 15 extra floats (structure confirmed); whether its first extra float is an intensity threshold (gtc2vcf) or unused (BeadArrayFiles) is still open. | partly resolved |
|
|
| K-001 | 2026-09-27 | IDAT MidBlock (plan U2) | In GSA v4 (BeadChip 48x4), HumanCNV370 (2x12) and 450k (12x8) IDATs the MidBlock count equals N and the section is 4 bytes per entry (int32, as in illuminaio). gtc2vcf reads uint8 but never uses the values. We infer the element size from the section length. | resolved for these chips + BeadChip 4x10 (Omni2.5, 2141 files) |
|