* simurg/core/brand.py: single source of the mark geometry, colours, favicon
and the report header band (scripts/brand now load it from there).
* HTML reports (QC report, compare dashboard, methylation QC): offline SVG
favicon, indigo header band with the mark, version and a sha256 of the data
shown; Noto Sans font stack; methylation plots in Okabe-Ito colours.
* README banner, guide logo, brand/blog links; branding brief updated to the
final identity; Noto (OFL) and Okabe-Ito entries in THIRD_PARTY_NOTICES;
CHANGELOG (Unreleased).
Unit tests + P5 QC oracle: 285 passed; ruff and mypy clean.
Frontal dotted simurg on a bead grid: gold body and wings, three-dot crown,
red / gold / green channel plumes with peacock eyes. Detailed (31x27), simple
(11x10) and micro (7x7) drawings; dark, light, gold, black, white versions;
outlined Noto Sans wordmark; favicon and app icons; README banner, social
preview, avatar, featured-image template, report corner mark, CLI banner.
Checks: WCAG contrast table, protan/deutan simulation, icon sizes, family
sheet. Blog featured images regenerated with the new mark for consistency.
Geometry shared in scripts/brand/simurg_mark.py.
Dotted simurg on a bead grid in the channel colors (red/green, gold bird,
indigo), one motif per post; generated by scripts/blog/make_featured_images.py.
- scripts/pgx/wgs_star_truth.py: star alleles from the 1000G 30x phased panel with the DB definitions
(wildcard-aware haplotype assignment; checked against GeT-RM on the demo CEPH individuals).
- validate_star_getrm.py: --all-genes, --individuals, --coverage (projection onto what the array can see);
replicate_concordance.py.
- 120 individuals x 3 GSA v1 arrays: array-resolvable 1637/1710 (95.7 %), top 1223/1800 (chip lacks many
defining variants), GeT-RM subset 28/35, replicates 99.2 % without the failed array (V2_pgx_star.md, K-029).
fetch_sources.sh downloads PharmVar per-gene zips; pgx db build adds core alleles for genes CPIC does not
define (normalised against the FASTA, *1 = reference). Masks re-derived with the new variants. GeT-RM:
CYP3A4 15/15, CYP2A6 7/9; all comparable genes 154/168 (CYP1A2/CYP2C8 definitions differ from GeT-RM's).
Sub-region CNs (p5, intron 2, exon 9) resolve full copies and *36/*68/*13 units; candidates carry the
tandem names, background alleles, and CPIC activity sums. GeT-RM: CYP2D6 13/15 top (NA12878 *3/*68+*4 and
NA11832 *1/*68+*4 exact), 15/15 within feasible; all genes 132/144.
- DB build resolves the PGX-007 tag variants from the manifest's probes; coverage over all gene kinds.
- Variant-based genes reported per variant (rs.. REF/ALT | Reference) with the CPIC lowest-activity diplotype;
tagging genes as positive/negative with the population caveat; GSTM1/GSTT1/UGT2B17 *0 from targeted CN.
- Case-insensitive phenotype keys (MT-RNR1). GeT-RM check extended to VKORC1 and the CN genes:
131/144 gene calls (docs/validation/V2_pgx_star.md).
- scripts/pgx/fetch_sources.sh: CPIC API tables (CC0) into a dated folder with version and hashes.
- simurg/pgx/db.py: CPIC HGVS locations and values -> left-normalised VCF variants (SNV, del, ins, dup,
repeats, delins, chrM); IUPAC values are core-allele wildcards; alleles without definitions are flagged.
- pipeline/pgx_db.py: deterministic SQLite (content-hash name), probe_map via the SNV VCF site mapper;
GSA v4 PGx: 22 genes, 1354 alleles, 5002 probe links; two builds byte-identical.
Mosaic fraction from the BAF split now subtracts the heterozygous BAF noise (median |BAF-0.5| of unsplit
hets = 0.6745 sigma_b), and a mosaic loss lacking its expected heterozygotes gets MF 1 and is promoted to a
constitutional DEL (CYTO-020). Adds scripts/v2/cn_tune.sh + cn_tune_report.py (GSE165845 rounds on the Pi).
The Pi now boots with cgroup_enable=memory cgroup_memory=1; the MemAvailable guard stays as a
second layer and as the fallback on systems without the memory controller.
scripts/pi_run.sh runs heavy jobs in their own process group and stops them when
MemAvailable drops below a floor (Raspberry Pi OS boots with cgroup_disable=memory, so a
systemd MemoryMax cap is not available); a 360-sample genotype run with cohort QC had
exhausted memory and the OOM killer took down the desktop session.
GSE165845 360 arrays: constitutional DEL/DUP calls 77/74 % in the NYGC SV truth, LOH
reproduced across replicates 74 %; mosaic calls are mostly false positives (12-23 %),
sensitivity for 25 kb-1 Mb events 7-13 % at QUAL >= 20. All samples had been flagged
SampleQuality by the summary logr_dev (cluster-file mismatch); the gate now uses the SD
of the LRR after the batch reference. Validation script: --ignore-filter.
cn call now records per-chromosome and sample statistics (percent het, LRR stats, BAF
deviations, robust LogR dev, call rate) in the sidecar; cn annotate writes
<sample>.cyto.json (CYTO-033) with ISCN microarray/traditional strings (CYTO-034), bands
and RefSeq gene lists (CYTO-035), LOH summaries, weighted median CN (CYTO-026); annotation
bundle from UCSC cytoBand + ncbiRefSeq (CYTO-036, IN-054). Tests for ISCN strings, gene
overlap, sidecar-missing error and an end-to-end call+annotate.
216 implanted events in 22 synthetic GSA samples (4 replicates per kind x fraction x
size; reference model from 30 other samples): constitutional DEL/DUP 16/16 at 5-20 Mb,
mosaic loss/gain/CN-LOH at 15 % / 5 Mb 12/12.
Found by the P10 fresh-VM run: `simurg inspect … | head` raised BrokenPipeError, reported as
an internal error, and stopped examples/gsa_demo.sh (set -o pipefail). run_guarded now flushes
stdout inside the guard and maps BrokenPipeError to ExitCode.SIGPIPE (141) without a message.
p10.sh: SKIP_DOCKER/SKIP_CONDA, clearer digest report; p10t.sh runs build+pip+tutorial only.
scripts/build_dist.sh: sdist + pure wheel + manylinux_2_28 kernel wheels (maturin --zig,
both architectures). Dockerfile (multi-stage, kernels included, non-root user).
packaging/conda/{simurg,simurg-core}. scripts/p10/digest.py: platform-independent content
digests of GTC/VCF/summary outputs. Cloud task p10 installs the artifacts on a fresh
Ubuntu VM and runs the tutorial with pip, Docker and conda installs. trial.sh picks the
arm64/amd64 image from the instance type and can run instances in parallel (PARALLEL=1).
Licence files ship in both wheels; Rust crate notices added.
Rust ports of the hot paths found by profiling: fused per-locus GenCall calling
and scoring, BAF/LRR, libm maps with CPython math semantics, both nearest-neighbour
searches of the normalization fit, and the BPM LocusEntry parser. Address
coverage is memoised per IDAT layout. SIMURG_PURE_PYTHON=1 forces the pure paths.
On the Pi: genotype_sample 3.4 s -> 1.1 s, 36-sample GSA run 131 s -> 46 s, BPM
read 4x faster; 36 GSA + 6 Omni2.5 GTCs and QC tables byte-identical to the pure
path; tests/unit/test_core_kernels.py checks every kernel on edge cases.
bench/profile_genotype.py, bench/scaling.py, cloud task p9 (thread scaling).
The download-only phase (EBI ~15 MB/s) left the CPU idle long enough for the
idle alarm to terminate the first run. Streams are now cut to the manifest
positions (±100 bp) and the 120 individuals while genotyping runs.
scripts/p8/prepare.py lays out the 360 GEO IDATs under their header barcode/position;
scripts/p8/validate_p8.py computes concordance, NRD, no-call rate, per-class and per-MAF
sensitivity/specificity/precision/F1 vs NYGC 30x and GIAB HG001, replicate concordance,
systematic/allele-swap site errors, per-chip batch table and duplicate detection.
Cloud userdata can skip the Omni2.5 archive (OMNI=0).
- compare/core: EXACT/NEAR/DISCREPANCY/UNKNOWN with the plan §16.3 tolerances (abs, float32 ULP), NaN = NaN,
locus classes (assay, variant type, chromosome group, intensity-only); fields absent on both sides are
not applicable
- compare/records: GTC vs GTC (per-locus arrays, transforms per bin, controls, sample statistics; metadata
listed, not counted) and VCF vs VCF (records keyed incl. ID; SNV/MNV/indel, merged, multi-allelic classes)
- simurg compare A B (files or folders paired by name): summary.json, discrepancies.tsv.gz (never truncated),
self-contained dashboard.html; --tolerance, --fail-on-discrepancy (exit 3)
- scripts/nightly.py + install_nightly.sh (crontab 02:30): unit tests, GSA vs gtc2vcf (GTC + VCF), VCF
regression vs golden, Omni2.5 subset vs vendor GTCs; first run PASS on every step
- tests/unit/test_compare.py; docs/validation/P7.md, README
- vendor comparison (1000 Genomes Omni2.5, AutoCall 1.6.2.2): every
computed field identical in 2138/2141 samples; the 3 others trace back
to the single-locus raw-intensity rule (K-009); sex rule 1 matches all
- AutoCall 1.6.2.x leaves GC10/GC50/#calls/#no-calls at 0 in GTC v3:
new `--preset autocall-1.6.2` (1.1.2 normalization, sex rule 1, GTC v3,
legacy statistics); the sample summary keeps the real statistics
- NaN sign canonicalised to the x86 pattern (K-014); report P4.md
- tests/unit/test_genotype.py: fuzzy memberships incl. NaN rules, calls
at cluster centres, intensity threshold, BAF/LRR interpolation and
NaN pattern, LogR-dev locus set, sex rules v1/v2, TOP-strand base
calls, invariant dates, end-to-end `simurg genotype` (summary schema,
pass-through columns, thread/layout determinism, overwrite refusal)
- tests/oracle/test_p4_genotype.py: 36 GSA samples x 2 versions
byte-identical to the oracle
- scripts/validate_p4_large.py + cloud task p4: compare every vendor
GTC field by field and as whole files; reports which sex rule matches
- fit.py: exact and binned nearest-neighbour searches use one distance
matrix per searched bin (first-minimum rule kept); fit 2.6 s -> 1.4 s
per GSA sample, still bit-identical to the oracle
- scripts/validate_p3_large.py: refits 1.2.0 and 1.1.2 transforms for
every vendor GTC and reports ULP distances per field
- cloud trial runs scripts/cloud/tasks/<TASK>.sh (TASK=p3 default;
p1p2 = previous validation + ingest)
- signal/normalization/numerics.py: vendor percentile, trimmean, MAD sigma
(old/new), bisquare robust fits, float32/float64 typing and sequential
sums reproduced exactly (ported from gtc2vcf, MIT)
- signal/normalization/fit.py: outlier removal, offset, rotation, shear,
scale, nn12/rr12, single-channel model for small Infinium I bins,
exact and binned nearest-neighbour sweeps; vendor quirks marked QUIRK
- signal/normalization/apply.py: affine transform and theta/R
- pipeline/normalize.py + CLI `simurg normalize`
- oracle GTCs rebuilt with -ffp-contract=off (aarch64 gcc fuses a*b+c by
default, shifting the last bits); the old FMA oracle differed by <=4 ULP
Result: transforms bit-identical to the oracle for all 36 GSA samples x
48 bins, for both normalization versions (3456/3456 bins, 0 ULP).
kemal