Commit Graph
12 Commits
Author SHA1 Message Date
kemal 8ee0c46bba Apply the brand across the repo: reports, README, guide
* simurg/core/brand.py: single source of the mark geometry, colours, favicon
  and the report header band (scripts/brand now load it from there).
* HTML reports (QC report, compare dashboard, methylation QC): offline SVG
  favicon, indigo header band with the mark, version and a sha256 of the data
  shown; Noto Sans font stack; methylation plots in Okabe-Ito colours.
* README banner, guide logo, brand/blog links; branding brief updated to the
  final identity; Noto (OFL) and Okabe-Ito entries in THIRD_PARTY_NOTICES;
  CHANGELOG (Unreleased).

Unit tests + P5 QC oracle: 285 passed; ruff and mypy clean.
2026-09-30 08:01:30 +01:00
kemal ed5e92b859 PGx: ClinPGx variant-based genes (PGX-008) and --guideline dpwg (PGX-025)
- fetch_sources.sh downloads ClinPGx clinicalVariants (CC BY-SA 4.0); pgx db build adds genotype-only
  variant-based genes (levels 1A-2B, level 3 for the parity list), rsIDs resolved from the manifest's probes:
  70 genes in the DB.
- pgx phenotype --guideline dpwg: CPIC mappings for CPIC genes (as ClinPGx curates DPWG), no phenotype for
  DPWG-only genes (flags dpwg_via_cpic_mapping / dpwg_mapping_not_available). K-029 updated.
2026-09-29 17:01:36 +01:00
kemal 9a39ae62fe PGx: PharmVar genes (CYP1A2, CYP2A6, CYP2A13, CYP2C8, CYP3A4), CYP2A6 *4 from CN (PGX-001/006)
fetch_sources.sh downloads PharmVar per-gene zips; pgx db build adds core alleles for genes CPIC does not
define (normalised against the FASTA, *1 = reference). Masks re-derived with the new variants. GeT-RM:
CYP3A4 15/15, CYP2A6 7/9; all comparable genes 154/168 (CYP1A2/CYP2C8 definitions differ from GeT-RM's).
2026-09-29 16:50:27 +01:00
kemal 0c5dd03ca8 PGx DB: pgx db build from CPIC (PGX-001..005, IN-053, CLI-035)
- scripts/pgx/fetch_sources.sh: CPIC API tables (CC0) into a dated folder with version and hashes.
- simurg/pgx/db.py: CPIC HGVS locations and values -> left-normalised VCF variants (SNV, del, ins, dup,
  repeats, delins, chrM); IUPAC values are core-allele wildcards; alleles without definitions are flagged.
- pipeline/pgx_db.py: deterministic SQLite (content-hash name), probe_map via the SNV VCF site mapper;
  GSA v4 PGx: 22 genes, 1354 alleles, 5002 probe links; two builds byte-identical.
2026-09-29 13:24:54 +01:00
kemal c3e0681b6e PGx CN: pgx copy-number train|call (CLI-031/032, PCN-001..017, IN-050), region BED v1, demo identities (K-028)
- simurg/pgx/cn.py: per-region CN mixture, mu_c = delta + gamma*log2((c+1)/3), free lower CN-0 component,
  Hardy-Weinberg-constrained weights over haplotype copy numbers, EM from a grid of label starts.
- pipeline/pgx_cn.py: model file (.simurg-pgxcn), batch rules (PCN-004), per-sample CN VCF + tbi, batch and
  sample summaries.
- The GSA v4 PGx demo arrays are 30 CEPH/1000G individuals (scripts/pgx/demo36_identity.tsv); against NYGC 30x
  SV RD_CN 224/240 region calls agree (GSTM1, UGT2B17 30/30), in-sample (docs/validation/V2_pgx_cn.md).
2026-09-29 13:16:19 +01:00
kemal 0bf5576049 v2 CN: cn annotate (JSON report, ISCN 2020, cytobands, genes) and the annotation bundle builder
cn call now records per-chromosome and sample statistics (percent het, LRR stats, BAF
deviations, robust LogR dev, call rate) in the sidecar; cn annotate writes
<sample>.cyto.json (CYTO-033) with ISCN microarray/traditional strings (CYTO-034), bands
and RefSeq gene lists (CYTO-035), LOH summaries, weighted median CN (CYTO-026); annotation
bundle from UCSC cytoBand + ncbiRefSeq (CYTO-036, IN-054). Tests for ISCN strings, gene
overlap, sidecar-missing error and an end-to-end call+annotate.
2026-09-29 01:37:56 +01:00
kemal 035e5ad66f P10 (WIP): release build script, Dockerfile, conda recipes, fresh-VM acceptance task
scripts/build_dist.sh: sdist + pure wheel + manylinux_2_28 kernel wheels (maturin --zig,
both architectures). Dockerfile (multi-stage, kernels included, non-root user).
packaging/conda/{simurg,simurg-core}. scripts/p10/digest.py: platform-independent content
digests of GTC/VCF/summary outputs. Cloud task p10 installs the artifacts on a fresh
Ubuntu VM and runs the tutorial with pip, Docker and conda installs. trial.sh picks the
arm64/amd64 image from the instance type and can run instances in parallel (PARALLEL=1).
Licence files ship in both wheels; Rust crate notices added.
2026-09-28 22:41:56 +01:00
kemal e65d2426bb P6: gtc-to-vcf (VCF/BCF + indexes), harmonisation, exports (plink, bedgraph, finalreport)
- formats/fasta: plain/BGZF FASTA, .fai/.gzi built identical to htslib, index check, build detection,
  contig aliases, mmap genome cache (IN-060..064)
- formats/bgzf, formats/bcf: pure-Python BGZF, tabix .tbi, BCF 2.2 and CSI (OUT-010, OUT-038, OUT-040)
- harmonize/sites: plus-strand mapping with audit rows, indel resolution from SourceSeq (port of GTCtoVCF,
  Apache-2.0) and left normalisation (OUT-014..021, OUT-030, OUT-042)
- harmonize/combine: OUT-031 genotype combination (ploidy-aware, Infinium I/II signal classes, OUT-037)
- pipeline/vcf: `simurg gtc-to-vcf` (per-sample VCF.gz+tbi, --merge cohort BCF+csi, audit parquet)
- pipeline/export: `simurg export bedgraph|finalreport|plink|locus-summary` (OUT-043, OUT-050..052, OUT-071)
- validation: 36 GSA samples vs gtc2vcf: 23.66 M identical calls, 0 unexplained differences; PLINK identical
  to plink2 on 637,397 variants; bcftools validates VCF/BCF and our indexes answer region queries correctly
- fetch_data.sh p6 (Ensembl GRCh38 with checksum), make_oracle_vcf.sh, validate_p6_vcf.py; K-022/K-023
2026-09-28 12:17:53 +01:00
kemal 1b6f834583 P5 validated on 2141 1000G samples; control-metric fixes (K-021)
- Omni2.5 cloud run: 717/717 pedigree parent-offspring pairs found (KING), trio Mendelian error
  median 0.031 %, sex vs pedigree 99.6 %, P4 preset re-check 2138/2141 byte-identical (K-015 closed)
- qc/controls: skip bead addresses absent from the IDAT; control missing only if none is on the chip
- QC-008 amended: signal / zero background = +Inf (passes), 0/0 = NaN (flagged); Inf in TSV/report
- plotly bundle stored gzip-compressed (repo grep stays clean: the DOM event name contains the
  forbidden substring)
- docs: validation P5 part 2, summaries, known_unknowns K-021
2026-09-28 10:25:32 +01:00
kemal 78e107aaa4 P5b: self-contained HTML QC report and QC table (simurg qc report)
- pipeline/qc_report: dataset discovery/labels/types (QC-041..043), robust TSV/CSV reading (QC-061),
  join + unmatched warning (QC-040), privacy options (QC-046), xlsx (4 sheets, conditional fills) / csv
  table (QC-045), byte-deterministic HTML with gzip+base64 embedded data (QC-044, QC-047)
- qc/report: template, CSS and JS app — Control Dashboard, Automated QC (table with sort/filter/search/
  paging/column menu/export, metric scatters with threshold lines), Sample Heatmaps (chip/plate,
  continuous/binary), Trend Analysis (grouped summary, Tukey box plots, counts, sex, PCA), thresholds modal
  (apply / download / upload YAML), linked selection across plots and table (QC-048..060)
- bundled plotly.js 2.35.2 (MIT); openpyxl dependency (MIT)
- tests: discovery/labels/errors/determinism/xlsx + headless-browser self-test
- known_unknowns K-020: remaining report deviations
2026-09-28 09:20:18 +01:00
kemal eb06e9de04 P5a: QC — control metrics, thresholds, status, contamination, relatedness, locus summary
- qc/controls: canonical control names (QC-002), bkg/offset (QC-003), 13 genotyping metrics (QC-004/005),
  missing/zero-divisor handling (QC-007/008); controls_raw.tsv / controls_qc.tsv (QC-009/011) incl. compat CSVs
- qc/thresholds + profiles: YAML schema, validation codes, precedence, shipped profiles, auto (QC-020..025)
- qc/status: functional PASS/FAIL/NA, control CLEAR/FLAGGED, flags (QC-030..033)
- qc/contamination: BAF regression; bit-compatible with gtc2vcf BAFregress (36/36, K-017), exact mode too
- qc/relatedness: KING-robust, duplicates, Mendelian errors, pedigree from sheet (IN-018)
- qc/locus_summary: OUT-072, bit-identical to BeadArrayFiles except ChiTest (K-018)
- qc/suggest: suggest-thresholds (QC-026)
- pipeline/qc: disk-backed cohort store; CLI qc controls|samples|contamination|relatedness|loci|suggest-thresholds
- genotype: control tables (OUT-070), het_rate, QC columns, cohort QC
- tests: unit + oracle (BAFregress plugin, BeadArrayFiles locus summary)
2026-09-28 08:35:10 +01:00
kemal 0a421dffc9 P0: project setup (packaging, CI, licences, data/oracle scripts)
- pyproject (uv/hatch, ruff, mypy strict, pytest markers), MIT LICENSE,
  THIRD_PARTY_NOTICES with verbatim BSD-2 (BeadArrayFiles) and MIT (gtc2vcf)
- CI workflow for Gitea/GitHub Actions: lint, type check, unit tests
- fetch_data.sh p0 now fetches the GSA v4 PGx demo IDATs (the GDA "Demo 36"
  bundle is a GenomeStudio project), unpacks archives, fixes pipefail bug
- make_oracle_gtcs.sh builds bcftools + gtc2vcf and writes oracle GTCs
- plan §17: corrected P0 dataset table
2026-09-27 16:56:39 +01:00