102 lines
7.0 KiB
Markdown
102 lines
7.0 KiB
Markdown
# Changelog
|
||
|
||
## Unreleased
|
||
|
||
**Documentation checks.** `scripts/docs/check_claims.py` checks the numeric claims of public texts against the
|
||
validation reports (code retrieves evidence, a TypeSafe System One model judges support). Its first run corrected the
|
||
manifest loading times (Omni2.5 about 18 s on a Pi, now measured in `docs/validation/P9.md`) and a demo timing
|
||
sentence in the README.
|
||
|
||
**Packaging.** The distributions are now `simurgarray` and `simurgarray-core` (the name `simurg` is taken on
|
||
PyPI); the `simurg` command and the `simurg` / `simurg_core` import names are unchanged. The conda recipe no longer pins
|
||
the kernel package to the Python package's version (it required `simurg-core == 2.0.0` while the kernels are 1.1.0).
|
||
Project URLs in the package metadata.
|
||
|
||
**Community.** `CONTRIBUTING.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, `CITATION.cff`, Gitea issue and pull request
|
||
templates; CI moved to `.gitea/workflows/`. Probe names in the P6 oracle examples are replaced by stable pseudonyms.
|
||
|
||
**Repository tidy-up.** Removed the Phase-0 research notes (`docs/research/`), the P9 benchmark scripts (`bench/`,
|
||
`scripts/cloud/tasks/p9*.sh`) and a vendor probe-name list that should never have been committed (also purged from
|
||
the git history).
|
||
|
||
**Branding.** A visual identity (`branding/`, guide in `branding/README.md`): a dotted simurg mark in detailed, simple
|
||
and micro drawings, wordmark and lockups, icons, README banner, social preview and featured-image templates, with
|
||
contrast and colour-vision checks. HTML reports (QC report, compare dashboard, methylation QC) get a favicon and a
|
||
header band with the mark, the software version and a checksum of the data they show. The mark geometry lives in
|
||
`simurg/core/brand.py`. Blog posts and featured images in `blog/`.
|
||
|
||
## 2.0.0 — 2026-09-30 (v2 modules)
|
||
|
||
The Python package is 2.0.0. The optional Rust kernels are unchanged and stay `simurg_core` 1.1.0.
|
||
|
||
**Methylation control QC** (`simurg methyl qc`, `simurg methyl convert-manifest`; MET-001 … MET-018): methylation
|
||
CSV manifests, the 21 control metrics, pOOBAH detection p-values, noob, linear/nonlinear dye bias, β and M-values,
|
||
summary tsv/xlsx/html with a β PCA, per-sample tables; methylation folders load in `simurg qc report`;
|
||
`qc controls --array-type methylation`. Checked against sesame (`docs/validation/V2_methylation.md`, K-026).
|
||
|
||
**Copy number, LOH and mosaicism** (`simurg cn build-model|call|annotate`; CYTO-001 … CYTO-042): GC-wave
|
||
correction, sex-aware batch reference, HMM with constitutional, LOH and mosaic states, event statistics and
|
||
filters, CNV VCF + segments + sidecar, ISCN 2020 annotation from UCSC tables; noise-normalised mosaic fractions and
|
||
complete-loss promotion (CYTO-041), per-probe LRR noise (CYTO-042). Validated on implanted events and on 360
|
||
GSE165845 arrays against the NYGC 30x SV callset (`docs/validation/V2_cn.md`, K-027).
|
||
|
||
**Pharmacogenomics** (`simurg pgx db build|coverage`, `pgx copy-number train|call`, `pgx star-allele call`,
|
||
`pgx phenotype`; PCN-001 … PCN-017, PGX-001 … PGX-035): targeted CN for CYP2D6 sub-regions, CYP2A6, GSTM1, GSTT1,
|
||
UGT2B17, SULT1A1, CYP2E1 (HWE-constrained compression mixture); a deterministic PGx DB from CPIC, PharmVar and
|
||
ClinPGx (70 genes); solver-free diplotype calling with array-equivalence classes, CYP2D6 structures (*5, xN,
|
||
*68+*4, *36+*10, *13), G6PD, HLA tags, variant-based and *0 genes; probe masks from 1000G 30x concordance; CPIC and
|
||
DPWG phenotypes; per-sample JSON and the TSV set. The GSA v4 PGx demo arrays are 30 CEPH/1000G individuals: region
|
||
CN 224/240 vs NYGC SV, gene calls 154/168 vs GeT-RM (`docs/validation/V2_pgx_cn.md`, `V2_pgx_star.md`, K-028, K-029).
|
||
|
||
**ArrayTrain** (`simurg train`; plan §12): trains EGT cluster files (v3, dbv9) from a cohort's GTCs with sex-aware
|
||
sample selection, BIC-selected 1/2/3-cluster θ mixtures, HWE-guided labelling, and missing clusters imputed from
|
||
cross-locus relations. On GSE165845 (chr22 vs 1000G 30x) it gives 99.44 % concordance at a 99.61 % call rate;
|
||
the vendor EGT gives 99.46 % at 99.15 % (`docs/validation/V2_arraytrain.md`, K-030).
|
||
|
||
## 1.1.0 — 2026-09-28 (P9, P10)
|
||
|
||
**Packaging (P10).** `scripts/build_dist.sh` (sdist, pure wheel, manylinux_2_28 kernel wheels for x86_64 and
|
||
aarch64), `Dockerfile`, conda recipes (`packaging/conda/`). Verified on fresh x86_64 and aarch64 VMs: pip, Docker
|
||
and conda installs give identical outputs (`docs/validation/P10.md`). A closed stdout pipe now exits 141 silently
|
||
instead of reporting an internal error.
|
||
|
||
**Performance (P9).**
|
||
|
||
Optional compiled kernels `rust/simurg-core` (PyO3): fused GenCall calling and BAF/LRR, libm maps with CPython
|
||
semantics, normalization nearest-neighbour searches, BPM parser. Byte-identical outputs to the pure-Python paths
|
||
(`SIMURG_PURE_PYTHON=1`); 78–80 GSA samples/min per Graviton3 core (pure Python 30), manifest loading 4× faster.
|
||
Address coverage memoised per IDAT layout; worker state pickled once. `bench/profile_genotype.py`,
|
||
`bench/scaling.py`. Details and scaling limits: `docs/validation/P9.md`.
|
||
|
||
## 1.0.0 — 2026-09-28
|
||
|
||
First release: the v1.0 goals of the master plan (§1.1), built and validated phase by phase (P0–P8).
|
||
|
||
**Formats (P1–P2).** Readers for IDAT (plain or gzipped; encrypted v1 refused), BPM, CSV manifest, EGT, GTC v3–v5
|
||
and sample sheets; `simurg inspect`; raw X/Y assembly into a canonical Parquet model (`simurg ingest`, resumable).
|
||
|
||
**Signal and calling (P3–P4).** Per-bin affine normalization 1.2.0 and 1.1.2; GenCall-equivalent calling from an
|
||
EGT with GenCall score; θ/R, BAF, LRR, LogR deviation, GC10/GC50, intensity percentiles, sex (rules 1 and 2);
|
||
GTC v5/v3 writer (`simurg genotype`). Presets `current` and `autocall-1.6.2`.
|
||
|
||
**QC (P5).** Control-probe metrics, threshold profiles and YAML, two-axis sample status and flags,
|
||
BAF-regression contamination, KING-robust relatedness and Mendel errors, locus summary, threshold suggestion, a
|
||
self-contained HTML QC report with xlsx/csv table (`simurg qc …`).
|
||
|
||
**Outputs (P6).** Plus-strand harmonisation with an audit trail; per-sample VCF.gz + tabix or cohort BCF + CSI
|
||
(`GT:GS:BAF:LRR`, optional INFO and intensity fields, merged multi-probe sites, indels from SourceSeq, SNV+indel
|
||
merging, auxiliary MNV loci); PLINK 1 `.bed` and PLINK 2 `.pgen`; BAF/LRR bedGraph; final report and genotype matrix.
|
||
|
||
**Testing (P7).** `simurg compare` (EXACT/NEAR/DISCREPANCY/UNKNOWN per field and locus class, HTML dashboard) and a
|
||
nightly differential run against gtc2vcf, GTCtoVCF and vendor GTCs.
|
||
|
||
**Validation (P8).** 360 GSA arrays of 120 1000 Genomes individuals vs NYGC 30x WGS (concordance 0.9934) and
|
||
GIAB HG001 (0.9985, non-ref F1 0.997); replicate concordance 0.9966. Vendor-equivalence: 2138/2141 Omni2.5 GTCs
|
||
byte-identical, zero unexplained VCF/PLINK differences vs gtc2vcf, GTCtoVCF and plink2.
|
||
|
||
Documentation: `docs/user_guide.md`, `examples/`, `docs/validation/P1.md … P8.md`, `docs/known_unknowns.md`,
|
||
`docs/messages.md`.
|
||
|
||
Known limitations: needs an existing cluster file (training is v2); no CN/LOH/PGx yet; largest tested cohort
|
||
2141 samples; pure NumPy speed.
|