scripts/p8/prepare.py lays out the 360 GEO IDATs under their header barcode/position;
scripts/p8/validate_p8.py computes concordance, NRD, no-call rate, per-class and per-MAF
sensitivity/specificity/precision/F1 vs NYGC 30x and GIAB HG001, replicate concordance,
systematic/allele-swap site errors, per-chip batch table and duplicate detection.
Cloud userdata can skip the Omni2.5 archive (OMNI=0).
- compare/core: EXACT/NEAR/DISCREPANCY/UNKNOWN with the plan §16.3 tolerances (abs, float32 ULP), NaN = NaN,
locus classes (assay, variant type, chromosome group, intensity-only); fields absent on both sides are
not applicable
- compare/records: GTC vs GTC (per-locus arrays, transforms per bin, controls, sample statistics; metadata
listed, not counted) and VCF vs VCF (records keyed incl. ID; SNV/MNV/indel, merged, multi-allelic classes)
- simurg compare A B (files or folders paired by name): summary.json, discrepancies.tsv.gz (never truncated),
self-contained dashboard.html; --tolerance, --fail-on-discrepancy (exit 3)
- scripts/nightly.py + install_nightly.sh (crontab 02:30): unit tests, GSA vs gtc2vcf (GTC + VCF), VCF
regression vs golden, Omni2.5 subset vs vendor GTCs; first run PASS on every step
- tests/unit/test_compare.py; docs/validation/P7.md, README
- gtc-to-vcf --info-fields ASSAY_TYPE,NORM_BIN,GENTRAIN,CLUSTER_SEP,ILMN_STRAND,PROBE_IDS (--egt) and
--extra-format X,Y,NORMX,NORMY,THETA,R, in per-sample VCFs and the --merge BCF (OUT-012, plan §8.3);
equal to gtc2vcf's values (THETA/R missing where both normalized intensities are 0, K-023)
- --merge-snv-indel (OUT-021, EDGE-013): alleles extended to the longest REF; SNV probes read the base at POS
- --auxiliary-loci (OUT-035, EDGE-015): rules (a)-(e), E_AUX_INVALID; rule (d) uses the last plus-strand base
for minus-strand designs (spec amended, as GTCtoVCF)
- export plink --format bed|pgen|both: PLINK 2 fixed-width .pgen/.pvar/.psam; plink2 reads it back to a .bed
byte-identical to ours
- cohort store keeps raw/normalized intensities for the BCF path; tests (unit + gtc2vcf oracle); docs
- Omni2.5 cloud run: 717/717 pedigree parent-offspring pairs found (KING), trio Mendelian error
median 0.031 %, sex vs pedigree 99.6 %, P4 preset re-check 2138/2141 byte-identical (K-015 closed)
- qc/controls: skip bead addresses absent from the IDAT; control missing only if none is on the chip
- QC-008 amended: signal / zero background = +Inf (passes), 0/0 = NaN (flagged); Inf in TSV/report
- plotly bundle stored gzip-compressed (repo grep stays clean: the DOM event name contains the
forbidden substring)
- docs: validation P5 part 2, summaries, known_unknowns K-021
- vendor comparison (1000 Genomes Omni2.5, AutoCall 1.6.2.2): every
computed field identical in 2138/2141 samples; the 3 others trace back
to the single-locus raw-intensity rule (K-009); sex rule 1 matches all
- AutoCall 1.6.2.x leaves GC10/GC50/#calls/#no-calls at 0 in GTC v3:
new `--preset autocall-1.6.2` (1.1.2 normalization, sex rule 1, GTC v3,
legacy statistics); the sample summary keeps the real statistics
- NaN sign canonicalised to the x86 pattern (K-014); report P4.md
- tests/unit/test_genotype.py: fuzzy memberships incl. NaN rules, calls
at cluster centres, intensity threshold, BAF/LRR interpolation and
NaN pattern, LogR-dev locus set, sex rules v1/v2, TOP-strand base
calls, invariant dates, end-to-end `simurg genotype` (summary schema,
pass-through columns, thread/layout determinism, overwrite refusal)
- tests/oracle/test_p4_genotype.py: 36 GSA samples x 2 versions
byte-identical to the oracle
- scripts/validate_p4_large.py + cloud task p4: compare every vendor
GTC field by field and as whole files; reports which sex rule matches
- genotype/clusters.py: EGT aligned to manifest order (float32 mean R)
- genotype/gencall.py: flair, six-case likelihood, priors/posteriors,
fuzzy sub-scores and score map, vectorised with the reference's
float/double evaluation order and libm transcendentals
- genotype/baf_lrr.py, sex.py (rules v1/v2), sample.py (IDAT -> Gtc)
- pipeline/inputs.py shared by ingest and genotype
- pipeline/genotype.py + CLI: GTC per sample, sample_summary.tsv/json
(OUT-060..063), run manifest
- GTCs from IDATs are byte-identical to the oracle for 36 GSA samples x
2 normalization versions (NaN sign canonicalised to the x86 pattern)
Normalization 1.1.2 reproduces all 109,191 bins of the 1000 Genomes
Omni2.5 GTCs (AutoCall 1.6.2.2) with 0 ULP difference, reserved fields
included; 1.2.0 and 1.1.2 are bit-identical to the no-FMA gtc2vcf oracle
on 36 GSA samples. AWS run: 33 min, ~$0.15, no resources left.
- fit.py: exact and binned nearest-neighbour searches use one distance
matrix per searched bin (first-minimum rule kept); fit 2.6 s -> 1.4 s
per GSA sample, still bit-identical to the oracle
- scripts/validate_p3_large.py: refits 1.2.0 and 1.1.2 transforms for
every vendor GTC and reports ULP distances per field
- cloud trial runs scripts/cloud/tasks/<TASK>.sh (TASK=p3 default;
p1p2 = previous validation + ingest)
- signal/normalization/numerics.py: vendor percentile, trimmean, MAD sigma
(old/new), bisquare robust fits, float32/float64 typing and sequential
sums reproduced exactly (ported from gtc2vcf, MIT)
- signal/normalization/fit.py: outlier removal, offset, rotation, shear,
scale, nn12/rr12, single-channel model for small Infinium I bins,
exact and binned nearest-neighbour sweeps; vendor quirks marked QUIRK
- signal/normalization/apply.py: affine transform and theta/R
- pipeline/normalize.py + CLI `simurg normalize`
- oracle GTCs rebuilt with -ffp-contract=off (aarch64 gcc fuses a*b+c by
default, shifting the last bits); the old FMA oracle differed by <=4 ULP
Result: transforms bit-identical to the oracle for all 36 GSA samples x
48 bins, for both normalization versions (3456/3456 bins, 0 ULP).
Raw X/Y and controls assembled from IDATs equal the vendor GTC arrays for
2138/2141 samples x 2.45M loci; 3 single-locus differences are recorded
as K-009 (probable per-channel missing rule in AutoCall 1.6.2.2). Channel
order X=Red confirmed (U1). `simurg ingest` processed all 2141 samples in
19 min (I/O-bound). AWS run cost ~$0.34 and left no resources.
- tests/oracle/test_p2_raw_xy.py: IDAT-assembled raw X/Y and controls
equal the arrays in oracle GTCs
- validate_p1_large.py: phase-P2 comparison against vendor GTCs with
diagnostics (channel swap, bead-count disagreements)
- cloud trial: also runs `simurg ingest` on all samples (time/memory),
disk 250 GB
All 2141 vendor-written GTC v3 files match BeadArrayFiles field by field
and re-encode byte-identically; all IDAT pairs parse; the 2.45M-locus BPM,
dbv-7 EGT and CSV manifest agree with BeadArrayFiles and each other.
Run on a self-destructing AWS spot instance (42 min, ~$0.20).
scripts/cloud/trial.sh launches one spot instance that validates the
parsers on the 1000 Genomes Omni2.5 set (2141 samples, vendor GTCs)
streamed from the public S3 bucket, uploads the summary and terminates
itself. Cost guards: spot price cap, hard time limit (shutdown =
terminate), EXIT-trap shutdown, CloudWatch idle alarm that terminates
from outside, disk deleted with the instance, monthly budget alert, and
an `audit` command listing anything still billable.
scripts/validate_p1_large.py compares every GTC with BeadArrayFiles,
checks byte-identical round-trips, IDAT pairs and BPM/EGT/CSV.
- pyproject (uv/hatch, ruff, mypy strict, pytest markers), MIT LICENSE,
THIRD_PARTY_NOTICES with verbatim BSD-2 (BeadArrayFiles) and MIT (gtc2vcf)
- CI workflow for Gitea/GitHub Actions: lint, type check, unit tests
- fetch_data.sh p0 now fetches the GSA v4 PGx demo IDATs (the GDA "Demo 36"
bundle is a GenomeStudio project), unpacks archives, fixes pipefail bug
- make_oracle_gtcs.sh builds bcftools + gtc2vcf and writes oracle GTCs
- plan §17: corrected P0 dataset table
Research notes R1-R4, documentation-study files A-E (source of the
functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use
neutral naming and current project decisions (plan §2).
Phase 0 deliverables: full technical specification and roadmap
(SIMURGARRAY_MASTER_PLAN.md), bootstrap script for reference
open-source repos and tiered dataset fetch script with checksums.
kemal