Commit Graph
91 Commits
Author SHA1 Message Date
kemal c71f1b6333 Apply the brand across the repo: reports, README, guide
* simurg/core/brand.py: single source of the mark geometry, colours, favicon
  and the report header band (scripts/brand now load it from there).
* HTML reports (QC report, compare dashboard, methylation QC): offline SVG
  favicon, indigo header band with the mark, version and a sha256 of the data
  shown; Noto Sans font stack; methylation plots in Okabe-Ito colours.
* README banner, guide logo, brand/blog links; branding brief updated to the
  final identity; Noto (OFL) and Okabe-Ito entries in THIRD_PARTY_NOTICES;
  CHANGELOG (Unreleased).

Unit tests + P5 QC oracle: 285 passed; ruff and mypy clean.
2026-09-30 08:01:30 +01:00
kemal aea1d459d9 Ignore editor lock files (.~lock.*#) 2026-09-30 07:52:07 +01:00
kemal 368fd30d11 branding: SimurgArray brand kit (mark, wordmark, lockups, icons, templates, guide)
Frontal dotted simurg on a bead grid: gold body and wings, three-dot crown,
red / gold / green channel plumes with peacock eyes. Detailed (31x27), simple
(11x10) and micro (7x7) drawings; dark, light, gold, black, white versions;
outlined Noto Sans wordmark; favicon and app icons; README banner, social
preview, avatar, featured-image template, report corner mark, CLI banner.
Checks: WCAG contrast table, protan/deutan simulation, icon sizes, family
sheet. Blog featured images regenerated with the new mark for consistency.
Geometry shared in scripts/brand/simurg_mark.py.
2026-09-30 07:51:52 +01:00
kemal e7ee009df3 blog: featured images (1200x630, ~90 KB) and a branding brief
Dotted simurg on a bead grid in the channel colors (red/green, gold bird,
indigo), one motif per post; generated by scripts/blog/make_featured_images.py.
2026-09-30 07:22:20 +01:00
kemal af39be1059 blog: five posts on SimurgArray (why, alternatives, impact, public release plan, usage and branding) 2026-09-30 06:24:25 +01:00
kemal 576c87d772 Release 2.0.0: v2 modules (ArrayTrain, CN/LOH, PGx, methylation QC)
ci / test (3.13) (push) Failing after 11s
ci / test (3.12) (push) Failing after 8s
User guide: ArrayTrain section, v2 commands in the table, v2 accuracy rows,
updated limitations. README status for v2.0.0. Version 2.0.0 (Python package);
the Rust kernels are unchanged and stay simurg_core 1.1.0. uv.lock committed
(uv lock --check clean).

Full suite on the Pi, file by file under the memory guard: 29 files exit 0
(unit: 282 passed), ruff and mypy clean.
2026-09-30 06:24:12 +01:00
kemal 8b0502107a test_p6_vcf: stream the gtc2vcf oracle output (OUT-012)
The INFO/extra-FORMAT comparison held ~700 000 gtc2vcf records as text plus two
dicts each (> 4.8 GB, stopped by the Pi memory guard). Now written to a temp
file and kept as compact tuples: peak 3.0 GB; all 5 tests pass.
2026-09-30 01:25:32 +01:00
kemal 47ee3c9f5d CN: final full-cohort cloud validation (GSE165845, CYTO-041/042 defaults)
360 arrays vs NYGC 30x SV: all samples OK (K-027 gate fix), PASS precision
DEL 44 % / DUP 57 % (first run 39 % / 34 % with filters ignored), 61 % without
the one excess_events array; rare DEL sensitivity 30 %; LOH replicated 85 %.
m7g.4xlarge spot, 17 min. docs/validation/V2_cn.md, K-027.
2026-09-29 23:04:16 +01:00
kemal e7ce242bd7 ArrayTrain: BIC-selected unbounded θ mixtures + HWE-guided labelling
Fixed θ windows mislabelled compressed/asymmetric loci (2-cluster loci: 1418 vs
186 errors). Now 1/2/3-component mixtures chosen by BIC, labelled by HWE
goodness of fit plus weak position and gap priors, split clusters merged.
GSE165845 chr22 vs 1000G 30x (240 arrays): concordance 99.44 % (v1 99.27 %,
vendor EGT 99.46 %), NRD 1.30 % (1.89 %, 1.23 %), call rate 99.61 %.
docs/validation/V2_arraytrain.md, K-030; plan §12, CLI-038.
2026-09-29 22:40:19 +01:00
kemal 2afd296776 train: loop variable typing (mypy) 2026-09-29 19:54:24 +01:00
kemal 35d1e08621 ArrayTrain (plan §12): simurg train — EGT cluster files from a cohort's GTCs
- simurg/train/clusters.py: vectorised per-locus θ mixture (AA/AB/BB + outlier component, θ-ordered bounds),
  cluster selection and merging, lone clusters homozygous, missing clusters imputed by cross-locus regression
  per assay type, cluster_sep and train (GenTrain-field) scores.
- formats/egt.py: write_egt (EGT v3, data block 9, round-trips through the reader).
- pipeline/train.py: normalised θ/R from GTC raw X/Y + transforms, sex-aware (X females, Y males, MT haploid),
  intensity-only loci as one R cluster.
- scripts/train/concordance_chr.py: array vs 1000G 30x on one chromosome.
Demo (36 GSA v4 PGx arrays, chr22 vs 1000G): trained 99.29 % / NRD 2.6 % vs vendor EGT 99.57 % / 1.55 %.
2026-09-29 19:53:53 +01:00
kemal 004739da8e PGx validation on GSE165845: WGS-derived star truth, 2-fold masks, array-resolvable concordance
- scripts/pgx/wgs_star_truth.py: star alleles from the 1000G 30x phased panel with the DB definitions
  (wildcard-aware haplotype assignment; checked against GeT-RM on the demo CEPH individuals).
- validate_star_getrm.py: --all-genes, --individuals, --coverage (projection onto what the array can see);
  replicate_concordance.py.
- 120 individuals x 3 GSA v1 arrays: array-resolvable 1637/1710 (95.7 %), top 1223/1800 (chip lacks many
  defining variants), GeT-RM subset 28/35, replicates 99.2 % without the failed array (V2_pgx_star.md, K-029).
2026-09-29 19:00:34 +01:00
kemal 2bd9773478 CHANGELOG: v2 CN/LOH/mosaicism and pharmacogenomics 2026-09-29 17:16:10 +01:00
kemal f0ea1eaefe docs: user guide sections for CN/LOH/mosaicism and pharmacogenomics 2026-09-29 17:15:43 +01:00
kemal f73c7f4391 tests: PGx demo oracle tests — definition sanity (PGX-023), limitations (PGX-009), determinism (PGX-029), JSON/TSV (PGX-028) 2026-09-29 17:15:09 +01:00
kemal c31187210b pgx_cn: PFB as float64 for the BAF evidence (mypy) 2026-09-29 17:11:50 +01:00
kemal 63d2cc56c8 PGx CN: scan-date batch warning (PCN-006), CN-consistent BAF evidence (PCN-010, --bafs-in-cn),
noise-outlier exclusion + robustness test (PCN-017/EDGE-026), platform from the CSV manifest (CLI-032)

Also: star-allele calls are checked for one manifest/build per run (PGX-011), duplications at the CN cap are
named xN (PGX-022), and tandem names attach supporting variants to their full copy.
2026-09-29 17:11:34 +01:00
kemal e1cfb3dfd0 PGx: ClinPGx variant-based genes (PGX-008) and --guideline dpwg (PGX-025)
- fetch_sources.sh downloads ClinPGx clinicalVariants (CC BY-SA 4.0); pgx db build adds genotype-only
  variant-based genes (levels 1A-2B, level 3 for the parity list), rsIDs resolved from the manifest's probes:
  70 genes in the DB.
- pgx phenotype --guideline dpwg: CPIC mappings for CPIC genes (as ClinPGx curates DPWG), no phenotype for
  DPWG-only genes (flags dpwg_via_cpic_mapping / dpwg_mapping_not_available). K-029 updated.
2026-09-29 17:01:36 +01:00
kemal dfba22496a PGx: PharmVar genes (CYP1A2, CYP2A6, CYP2A13, CYP2C8, CYP3A4), CYP2A6 *4 from CN (PGX-001/006)
fetch_sources.sh downloads PharmVar per-gene zips; pgx db build adds core alleles for genes CPIC does not
define (normalised against the FASTA, *1 = reference). Masks re-derived with the new variants. GeT-RM:
CYP3A4 15/15, CYP2A6 7/9; all comparable genes 154/168 (CYP1A2/CYP2C8 definitions differ from GeT-RM's).
2026-09-29 16:50:27 +01:00
kemal 46422b8de4 PGx: CYP2D6 structures from sub-region CN — *68+*4, *36+*10, *13 (PGX-013/018)
Sub-region CNs (p5, intron 2, exon 9) resolve full copies and *36/*68/*13 units; candidates carry the
tandem names, background alleles, and CPIC activity sums. GeT-RM: CYP2D6 13/15 top (NA12878 *3/*68+*4 and
NA11832 *1/*68+*4 exact), 15/15 within feasible; all genes 132/144.
2026-09-29 16:29:54 +01:00
kemal e567d12684 PGx: all CPIC gene kinds — G6PD, variant-based, HLA tags, *0 CN genes (PGX-006/007/021)
- DB build resolves the PGX-007 tag variants from the manifest's probes; coverage over all gene kinds.
- Variant-based genes reported per variant (rs.. REF/ALT | Reference) with the CPIC lowest-activity diplotype;
  tagging genes as positive/negative with the population caveat; GSTM1/GSTT1/UGT2B17 *0 from targeted CN.
- Case-insensitive phenotype keys (MT-RNR1). GeT-RM check extended to VKORC1 and the CN genes:
  131/144 gene calls (docs/validation/V2_pgx_star.md).
2026-09-29 16:24:33 +01:00
kemal abea028818 PGx: full call detail (PGX-016..020, 027/028, 031), pgx coverage (PGX-009/035), full HGVS
- supporting_variants.tsv, unresolved_variants.tsv, variants_not_covered.tsv with PGX-026 metadata; per-allele
  supporting/unresolved sites for both alleles of every candidate; variant_change vs top, distant_alternative,
  unexplained_variants, alleles_impacted_by_no_call, alleles_tested.
- simurg pgx coverage: callable / not_covered / masked / equivalent_to per allele and manifest.
- HGVS on GRCh38 RefSeq accessions (NC_...:g.).
2026-09-29 16:16:39 +01:00
kemal 9d3242828e PGx phenotypes: pgx phenotype|annotate (PGX-024..028, CLI-034)
CPIC lookups by allele function counts or activity values (duplications N x activity, *5/*0 = 0, total-score
fallback), Indeterminate when feasible solutions disagree or an allele is not in the guideline; per-sample
simurg.pgx/1 JSON and diplotype_annotations.tsv with PGX-026 metadata lines.
2026-09-29 16:03:27 +01:00
kemal c2b45f8a76 derive_masks: keep pysam's downloaded tabix indexes in data/pgx/cache, not the working directory 2026-09-29 16:00:25 +01:00
kemal 194c087b24 PGx star alleles: pgx star-allele call (PGX-011..021), probe masks vs 1000G, GeT-RM validation (K-029)
- simurg/pgx/star.py: solver-free diplotype enumeration incl. CN forms (deletion, xN), wildcards, array-equivalence
  classes, raw_score/score with CPIC frequency priors; calibrated error model (spec PGX-012 amended).
- pipeline/pgx_star.py: DB gene models, SNV + PGx CN VCF evidence, masks, diplotype_summary.tsv + pgx_calls.json.
- scripts/pgx: derive_masks.py (array vs 1000G 30x over HTTPS, GS calibration), getrm_truth.py,
  validate_star_getrm.py.
- GSA v4 PGx demo vs GeT-RM (16 CEPH): 100/109 gene calls; CYP2D6/2C19/2C9 41/45 (docs/validation/V2_pgx_star.md).
2026-09-29 14:08:03 +01:00
kemal 7e9ff28501 PGx DB: pgx db build from CPIC (PGX-001..005, IN-053, CLI-035)
- scripts/pgx/fetch_sources.sh: CPIC API tables (CC0) into a dated folder with version and hashes.
- simurg/pgx/db.py: CPIC HGVS locations and values -> left-normalised VCF variants (SNV, del, ins, dup,
  repeats, delins, chrM); IUPAC values are core-allele wildcards; alleles without definitions are flagged.
- pipeline/pgx_db.py: deterministic SQLite (content-hash name), probe_map via the SNV VCF site mapper;
  GSA v4 PGx: 22 genes, 1354 alleles, 5002 probe links; two builds byte-identical.
2026-09-29 13:24:54 +01:00
kemal 7dc1fbc014 PGx CN: pgx copy-number train|call (CLI-031/032, PCN-001..017, IN-050), region BED v1, demo identities (K-028)
- simurg/pgx/cn.py: per-region CN mixture, mu_c = delta + gamma*log2((c+1)/3), free lower CN-0 component,
  Hardy-Weinberg-constrained weights over haplotype copy numbers, EM from a grid of label starts.
- pipeline/pgx_cn.py: model file (.simurg-pgxcn), batch rules (PCN-004), per-sample CN VCF + tbi, batch and
  sample summaries.
- The GSA v4 PGx demo arrays are 30 CEPH/1000G individuals (scripts/pgx/demo36_identity.tsv); against NYGC 30x
  SV RD_CN 224/240 region calls agree (GSTM1, UGT2B17 30/30), in-sample (docs/validation/V2_pgx_cn.md).
2026-09-29 13:16:19 +01:00
kemal e4870f0869 V2 CN: CYTO-042 results (precision up, mosaic calls halved); pi_run reports process memory without page cache 2026-09-29 12:20:40 +01:00
kemal 515167a990 cn build-model: compute the per-probe LRR spread in column chunks (peak memory hit the 5 GB cap on 120 GSA arrays) 2026-09-29 12:11:38 +01:00
kemal 8dba275b87 CN: per-probe LRR noise from the batch reference (CYTO-042), --probe-lrr-sd/--no-probe-lrr-sd 2026-09-29 12:00:04 +01:00
kemal 643db76891 cn_validate_truth: sensitivity by carrier frequency (rare < 5 % is the primary metric, K-027) 2026-09-29 11:57:58 +01:00
kemal 717a0aad76 V2 CN validation: Pi tuning rounds, CYTO-041 effect, common-CNV sensitivity limit (K-027) 2026-09-29 11:11:14 +01:00
kemal aa3031a18a CN: MF split normalised by per-probe reference BAF noise; genotype-based complete loss (CYTO-041)
The mosaic BAF split is now estimated over heterozygous calls against each probe's expected noise (the
reference heterozygous spread scaled to the sample), inverting the median of |N(+-d, sigma)|. A mosaic loss
lacking the heterozygous calls its PFB predicts gets MF 1 and is promoted (CYTO-020).
2026-09-29 10:56:21 +01:00
kemal 4315f37f1f CN: noise-corrected BAF split and complete-loss MF (CYTO-041); Pi tuning harness
Mosaic fraction from the BAF split now subtracts the heterozygous BAF noise (median |BAF-0.5| of unsplit
hets = 0.6745 sigma_b), and a mosaic loss lacking its expected heterozygotes gets MF 1 and is promoted to a
constitutional DEL (CYTO-020). Adds scripts/v2/cn_tune.sh + cn_tune_report.py (GSE165845 rounds on the Pi).
2026-09-29 10:30:12 +01:00
kemal dc0dbedfc2 pi_run: hard per-job memory cap via a systemd scope (MemoryMax) when the memory cgroup is enabled
The Pi now boots with cgroup_enable=memory cgroup_memory=1; the MemAvailable guard stays as a
second layer and as the fallback on systems without the memory controller.
2026-09-29 08:24:29 +01:00
kemal 265482a556 CN: per-probe heterozygous BAF spread from the batch reference; --mosaic-prior/--switch-distance; Pi memory guard
scripts/pi_run.sh runs heavy jobs in their own process group and stops them when
MemAvailable drops below a floor (Raspberry Pi OS boots with cgroup_disable=memory, so a
systemd MemoryMax cap is not available); a 360-sample genotype run with cohort QC had
exhausted memory and the OOM killer took down the desktop session.
2026-09-29 07:56:12 +01:00
kemal a5728d84e3 v2 CN: real-data validation vs NYGC 30x SVs (interim), noise gate on normalised LRR (K-027)
GSE165845 360 arrays: constitutional DEL/DUP calls 77/74 % in the NYGC SV truth, LOH
reproduced across replicates 74 %; mosaic calls are mostly false positives (12-23 %),
sensitivity for 25 kb-1 Mb events 7-13 % at QUAL >= 20. All samples had been flagged
SampleQuality by the summary logr_dev (cluster-file mismatch); the gate now uses the SD
of the LRR after the batch reference. Validation script: --ignore-filter.
2026-09-29 07:42:06 +01:00
kemal 726fe045f5 test: packaged CN data files 2026-09-29 07:07:03 +01:00
kemal e4be579bfe Track the packaged CN data files (the data/ ignore rule matched src/simurg/cn/data)
Found by the CN cloud validation: mask_GRCh3{7,8}.bed and cytoBand_GRCh3{7,8}.txt.gz were
never committed nor shipped in the wheel. The ignore rule now covers only the top-level data/.
2026-09-29 07:06:47 +01:00
kemal bc24148e31 v2 CN: truth validation script (NYGC 30x SV freeze V3) and cloud task cnv 2026-09-29 01:39:49 +01:00
kemal d3097dd027 v2 CN: cn annotate (JSON report, ISCN 2020, cytobands, genes) and the annotation bundle builder
cn call now records per-chromosome and sample statistics (percent het, LRR stats, BAF
deviations, robust LogR dev, call rate) in the sidecar; cn annotate writes
<sample>.cyto.json (CYTO-033) with ISCN microarray/traditional strings (CYTO-034), bands
and RefSeq gene lists (CYTO-035), LOH summaries, weighted median CN (CYTO-026); annotation
bundle from UCSC cytoBand + ncbiRefSeq (CYTO-036, IN-054). Tests for ISCN strings, gene
overlap, sidecar-missing error and an end-to-end call+annotate.
2026-09-29 01:37:56 +01:00
kemal 1255f37a16 v2 CN: unit tests (GC windows, PFB, GC-wave, smoothing, HMM DEL/LOH, effective size, thresholds, mask, CLI VCF rules) 2026-09-29 01:32:56 +01:00
kemal 2ad3a044a0 v2 CN: synthetic implant evaluation (CYTO-022); reference GTC discovery follows symlinks
216 implanted events in 22 synthetic GSA samples (4 replicates per kind x fraction x
size; reference model from 30 other samples): constitutional DEL/DUP 16/16 at 5-20 Mb,
mosaic loss/gain/CN-LOH at 15 % / 5 Mb 12/12.
2026-09-29 01:30:07 +01:00
kemal e76b87939b v2 CN/LOH (WIP): cn build-model and cn call
Cyto model (.simurg-cyto: GC 1 kb/100 kb/1 Mb, PFB from the cluster file, optional
sex-aware batch reference from normal GTCs; IN-051, CYTO-038); GC-wave correction
(CYTO-003); smoothing (CYTO-004); HMM with constitutional, gain-LOH, mosaic and haploid
states, forward-backward QUAL and Viterbi segments (CYTO-005/012); event statistics,
effective size, post-filters, per-type thresholds, promotion/LowFraction, built-in
centromere mask from UCSC cytoBand (CYTO-006..021); sex resolution (CYTO-024), ploidy;
CNV VCF with bgzip+tabix (CYTO-028..031), segments parquet, cn_sample_summary.tsv and
the cn_call_manifest.json sidecar.
2026-09-29 01:13:51 +01:00
kemal 579009271e v2 methylation QC validated on minfiDataEPIC and against sesame (K-026)
beta r 0.99998 vs sesame 1.28.1 (max |dbeta| 0.011), detection calls 99.1% identical,
noob within 0.5% median; replicate r 0.997; all 21 control metrics pass. Spec deviations
recorded (5-field control lines, pOOBAH formula, minfi sheet keys). User guide section.
2026-09-29 00:41:54 +01:00
kemal b8695c835e methylation: combine_neg option; oracle compares both background variants and logs sesame's negative controls 2026-09-29 00:36:44 +01:00
kemal 9dd80c8179 sesame oracle: compare without bead-count masking on both sides 2026-09-29 00:29:25 +01:00
kemal 851ce2f242 sesame oracle: cache EPIC annotation up front 2026-09-29 00:23:24 +01:00
kemal ddd2fb32c0 sesame oracle: explicit platform, cache fallback 2026-09-29 00:18:17 +01:00
kemal 256e28c393 v2 methylation: sesame oracle scripts and cloud task 2026-09-29 00:14:18 +01:00