kemal
7d2dbe939e
Graviton3 c7g.8xlarge, 360 GSE165845 arrays: 360/360 GTCs identical between the Rust kernels and the pure-Python paths; 1 worker 78-80 samples/min (pure Python 29.6). Scaling 1->8 workers efficiency 0.87-0.90; above that the 125 MB/s gp3 disk caps the run (~620 samples/min) and, on a RAM disk, a fixed per-worker start-up cost limits a 360-sample batch to efficiency 0.48 at 32 workers (K-025). Worker state is pickled once. docs/validation/P9.md, result JSONs, guide/README/CHANGELOG updates.
42 lines
2.5 KiB
Markdown
42 lines
2.5 KiB
Markdown
# Changelog
|
||
|
||
## Unreleased (P9)
|
||
|
||
Optional compiled kernels `rust/simurg-core` (PyO3): fused GenCall calling and BAF/LRR, libm maps with CPython
|
||
semantics, normalization nearest-neighbour searches, BPM parser. Byte-identical outputs to the pure-Python paths
|
||
(`SIMURG_PURE_PYTHON=1`); 78–80 GSA samples/min per Graviton3 core (pure Python 30), manifest loading 4× faster.
|
||
Address coverage memoised per IDAT layout; worker state pickled once. `bench/profile_genotype.py`,
|
||
`bench/scaling.py`. Details and scaling limits: `docs/validation/P9.md`.
|
||
|
||
## 1.0.0 — 2026-09-28
|
||
|
||
First release: the v1.0 goals of the master plan (§1.1), built and validated phase by phase (P0–P8).
|
||
|
||
**Formats (P1–P2).** Readers for IDAT (plain or gzipped; encrypted v1 refused), BPM, CSV manifest, EGT, GTC v3–v5
|
||
and sample sheets; `simurg inspect`; raw X/Y assembly into a canonical Parquet model (`simurg ingest`, resumable).
|
||
|
||
**Signal and calling (P3–P4).** Per-bin affine normalization 1.2.0 and 1.1.2; GenCall-equivalent calling from an
|
||
EGT with GenCall score; θ/R, BAF, LRR, LogR deviation, GC10/GC50, intensity percentiles, sex (rules 1 and 2);
|
||
GTC v5/v3 writer (`simurg genotype`). Presets `current` and `autocall-1.6.2`.
|
||
|
||
**QC (P5).** Control-probe metrics, threshold profiles and YAML, two-axis sample status and flags,
|
||
BAF-regression contamination, KING-robust relatedness and Mendel errors, locus summary, threshold suggestion, a
|
||
self-contained HTML QC report with xlsx/csv table (`simurg qc …`).
|
||
|
||
**Outputs (P6).** Plus-strand harmonisation with an audit trail; per-sample VCF.gz + tabix or cohort BCF + CSI
|
||
(`GT:GS:BAF:LRR`, optional INFO and intensity fields, merged multi-probe sites, indels from SourceSeq, SNV+indel
|
||
merging, auxiliary MNV loci); PLINK 1 `.bed` and PLINK 2 `.pgen`; BAF/LRR bedGraph; final report and genotype matrix.
|
||
|
||
**Testing (P7).** `simurg compare` (EXACT/NEAR/DISCREPANCY/UNKNOWN per field and locus class, HTML dashboard) and a
|
||
nightly differential run against gtc2vcf, GTCtoVCF and vendor GTCs.
|
||
|
||
**Validation (P8).** 360 GSA arrays of 120 1000 Genomes individuals vs NYGC 30x WGS (concordance 0.9934) and
|
||
GIAB HG001 (0.9985, non-ref F1 0.997); replicate concordance 0.9966. Vendor-equivalence: 2138/2141 Omni2.5 GTCs
|
||
byte-identical, zero unexplained VCF/PLINK differences vs gtc2vcf, GTCtoVCF and plink2.
|
||
|
||
Documentation: `docs/user_guide.md`, `examples/`, `docs/validation/P1.md … P8.md`, `docs/known_unknowns.md`,
|
||
`docs/messages.md`.
|
||
|
||
Known limitations: needs an existing cluster file (training is v2); no CN/LOH/PGx yet; largest tested cohort
|
||
2141 samples; pure NumPy speed.
|