The INFO/extra-FORMAT comparison held ~700 000 gtc2vcf records as text plus two
dicts each (> 4.8 GB, stopped by the Pi memory guard). Now written to a temp
file and kept as compact tuples: peak 3.0 GB; all 5 tests pass.
- gtc-to-vcf --info-fields ASSAY_TYPE,NORM_BIN,GENTRAIN,CLUSTER_SEP,ILMN_STRAND,PROBE_IDS (--egt) and
--extra-format X,Y,NORMX,NORMY,THETA,R, in per-sample VCFs and the --merge BCF (OUT-012, plan §8.3);
equal to gtc2vcf's values (THETA/R missing where both normalized intensities are 0, K-023)
- --merge-snv-indel (OUT-021, EDGE-013): alleles extended to the longest REF; SNV probes read the base at POS
- --auxiliary-loci (OUT-035, EDGE-015): rules (a)-(e), E_AUX_INVALID; rule (d) uses the last plus-strand base
for minus-strand designs (spec amended, as GTCtoVCF)
- export plink --format bed|pgen|both: PLINK 2 fixed-width .pgen/.pvar/.psam; plink2 reads it back to a .bed
byte-identical to ours
- cohort store keeps raw/normalized intensities for the BCF path; tests (unit + gtc2vcf oracle); docs
- genotype/clusters.py: EGT aligned to manifest order (float32 mean R)
- genotype/gencall.py: flair, six-case likelihood, priors/posteriors,
fuzzy sub-scores and score map, vectorised with the reference's
float/double evaluation order and libm transcendentals
- genotype/baf_lrr.py, sex.py (rules v1/v2), sample.py (IDAT -> Gtc)
- pipeline/inputs.py shared by ingest and genotype
- pipeline/genotype.py + CLI: GTC per sample, sample_summary.tsv/json
(OUT-060..063), run manifest
- GTCs from IDATs are byte-identical to the oracle for 36 GSA samples x
2 normalization versions (NaN sign canonicalised to the x86 pattern)
- signal/normalization/numerics.py: vendor percentile, trimmean, MAD sigma
(old/new), bisquare robust fits, float32/float64 typing and sequential
sums reproduced exactly (ported from gtc2vcf, MIT)
- signal/normalization/fit.py: outlier removal, offset, rotation, shear,
scale, nn12/rr12, single-channel model for small Infinium I bins,
exact and binned nearest-neighbour sweeps; vendor quirks marked QUIRK
- signal/normalization/apply.py: affine transform and theta/R
- pipeline/normalize.py + CLI `simurg normalize`
- oracle GTCs rebuilt with -ffp-contract=off (aarch64 gcc fuses a*b+c by
default, shifting the last bits); the old FMA oracle differed by <=4 ULP
Result: transforms bit-identical to the oracle for all 36 GSA samples x
48 bins, for both normalization versions (3456/3456 bins, 0 ULP).
- tests/oracle/test_p2_raw_xy.py: IDAT-assembled raw X/Y and controls
equal the arrays in oracle GTCs
- validate_p1_large.py: phase-P2 comparison against vendor GTCs with
diagnostics (channel swap, bead-count disagreements)
- cloud trial: also runs `simurg ingest` on all samples (time/memory),
disk 250 GB
kemal