SimurgArray/CHANGELOG.md

Changelog

Unreleased

Repository tidy-up. Removed the Phase-0 research notes (docs/research/), the P9 benchmark scripts (bench/, scripts/cloud/tasks/p9*.sh) and a vendor probe-name list that should never have been committed (also purged from the git history).

Branding. A visual identity (branding/, guide in branding/README.md): a dotted simurg mark in detailed, simple and micro drawings, wordmark and lockups, icons, README banner, social preview and featured-image templates, with contrast and colour-vision checks. HTML reports (QC report, compare dashboard, methylation QC) get a favicon and a header band with the mark, the software version and a checksum of the data they show. The mark geometry lives in simurg/core/brand.py. Blog posts and featured images in blog/.

2.0.0 — 2026-09-30 (v2 modules)

The Python package is 2.0.0. The optional Rust kernels are unchanged and stay simurg_core 1.1.0.

Methylation control QC (simurg methyl qc, simurg methyl convert-manifest; MET-001 … MET-018): methylation CSV manifests, the 21 control metrics, pOOBAH detection p-values, noob, linear/nonlinear dye bias, β and M-values, summary tsv/xlsx/html with a β PCA, per-sample tables; methylation folders load in simurg qc report; qc controls --array-type methylation. Checked against sesame (docs/validation/V2_methylation.md, K-026).

Copy number, LOH and mosaicism (simurg cn build-model|call|annotate; CYTO-001 … CYTO-042): GC-wave correction, sex-aware batch reference, HMM with constitutional, LOH and mosaic states, event statistics and filters, CNV VCF + segments + sidecar, ISCN 2020 annotation from UCSC tables; noise-normalised mosaic fractions and complete-loss promotion (CYTO-041), per-probe LRR noise (CYTO-042). Validated on implanted events and on 360 GSE165845 arrays against the NYGC 30x SV callset (docs/validation/V2_cn.md, K-027).

Pharmacogenomics (simurg pgx db build|coverage, pgx copy-number train|call, pgx star-allele call, pgx phenotype; PCN-001 … PCN-017, PGX-001 … PGX-035): targeted CN for CYP2D6 sub-regions, CYP2A6, GSTM1, GSTT1, UGT2B17, SULT1A1, CYP2E1 (HWE-constrained compression mixture); a deterministic PGx DB from CPIC, PharmVar and ClinPGx (70 genes); solver-free diplotype calling with array-equivalence classes, CYP2D6 structures (*5, xN, *68+*4, *36+*10, *13), G6PD, HLA tags, variant-based and *0 genes; probe masks from 1000G 30x concordance; CPIC and DPWG phenotypes; per-sample JSON and the TSV set. The GSA v4 PGx demo arrays are 30 CEPH/1000G individuals: region CN 224/240 vs NYGC SV, gene calls 154/168 vs GeT-RM (docs/validation/V2_pgx_cn.md, V2_pgx_star.md, K-028, K-029).

ArrayTrain (simurg train; plan §12): trains EGT cluster files (v3, dbv9) from a cohort's GTCs with sex-aware sample selection, BIC-selected 1/2/3-cluster θ mixtures, HWE-guided labelling, and missing clusters imputed from cross-locus relations. On GSE165845 (chr22 vs 1000G 30x) it gives 99.44 % concordance at a 99.61 % call rate; the vendor EGT gives 99.46 % at 99.15 % (docs/validation/V2_arraytrain.md, K-030).

1.1.0 — 2026-09-28 (P9, P10)

Packaging (P10). scripts/build_dist.sh (sdist, pure wheel, manylinux_2_28 kernel wheels for x86_64 and aarch64), Dockerfile, conda recipes (packaging/conda/). Verified on fresh x86_64 and aarch64 VMs: pip, Docker and conda installs give identical outputs (docs/validation/P10.md). A closed stdout pipe now exits 141 silently instead of reporting an internal error.

Performance (P9).

Optional compiled kernels rust/simurg-core (PyO3): fused GenCall calling and BAF/LRR, libm maps with CPython semantics, normalization nearest-neighbour searches, BPM parser. Byte-identical outputs to the pure-Python paths (SIMURG_PURE_PYTHON=1); 78–80 GSA samples/min per Graviton3 core (pure Python 30), manifest loading 4× faster. Address coverage memoised per IDAT layout; worker state pickled once. bench/profile_genotype.py, bench/scaling.py. Details and scaling limits: docs/validation/P9.md.

1.0.0 — 2026-09-28

First release: the v1.0 goals of the master plan (§1.1), built and validated phase by phase (P0–P8).

Formats (P1–P2). Readers for IDAT (plain or gzipped; encrypted v1 refused), BPM, CSV manifest, EGT, GTC v3–v5 and sample sheets; simurg inspect; raw X/Y assembly into a canonical Parquet model (simurg ingest, resumable).

Signal and calling (P3–P4). Per-bin affine normalization 1.2.0 and 1.1.2; GenCall-equivalent calling from an EGT with GenCall score; θ/R, BAF, LRR, LogR deviation, GC10/GC50, intensity percentiles, sex (rules 1 and 2); GTC v5/v3 writer (simurg genotype). Presets current and autocall-1.6.2.

QC (P5). Control-probe metrics, threshold profiles and YAML, two-axis sample status and flags, BAF-regression contamination, KING-robust relatedness and Mendel errors, locus summary, threshold suggestion, a self-contained HTML QC report with xlsx/csv table (simurg qc …).

Outputs (P6). Plus-strand harmonisation with an audit trail; per-sample VCF.gz + tabix or cohort BCF + CSI (GT:GS:BAF:LRR, optional INFO and intensity fields, merged multi-probe sites, indels from SourceSeq, SNV+indel merging, auxiliary MNV loci); PLINK 1 .bed and PLINK 2 .pgen; BAF/LRR bedGraph; final report and genotype matrix.

Testing (P7). simurg compare (EXACT/NEAR/DISCREPANCY/UNKNOWN per field and locus class, HTML dashboard) and a nightly differential run against gtc2vcf, GTCtoVCF and vendor GTCs.

Validation (P8). 360 GSA arrays of 120 1000 Genomes individuals vs NYGC 30x WGS (concordance 0.9934) and GIAB HG001 (0.9985, non-ref F1 0.997); replicate concordance 0.9966. Vendor-equivalence: 2138/2141 Omni2.5 GTCs byte-identical, zero unexplained VCF/PLINK differences vs gtc2vcf, GTCtoVCF and plink2.

Documentation: docs/user_guide.md, examples/, docs/validation/P1.md … P8.md, docs/known_unknowns.md, docs/messages.md.

Known limitations: needs an existing cluster file (training is v2); no CN/LOH/PGx yet; largest tested cohort 2141 samples; pure NumPy speed.