kemal
e76b87939b
v2 CN/LOH (WIP): cn build-model and cn call
...
Cyto model (.simurg-cyto: GC 1 kb/100 kb/1 Mb, PFB from the cluster file, optional
sex-aware batch reference from normal GTCs; IN-051, CYTO-038); GC-wave correction
(CYTO-003); smoothing (CYTO-004); HMM with constitutional, gain-LOH, mosaic and haploid
states, forward-backward QUAL and Viterbi segments (CYTO-005/012); event statistics,
effective size, post-filters, per-type thresholds, promotion/LowFraction, built-in
centromere mask from UCSC cytoBand (CYTO-006..021); sex resolution (CYTO-024), ploidy;
CNV VCF with bgzip+tabix (CYTO-028..031), segments parquet, cn_sample_summary.tsv and
the cn_call_manifest.json sidecar.
2026-09-29 01:13:51 +01:00
kemal
1fc27a866a
v2 methylation control QC (WIP): simurg methyl qc|convert-manifest (MET-001..018)
...
Methylation CSV manifest reader (MET-002/003), raw M/U assembly with bead masking
(MET-005), pOOBAH detection p-values, noob background correction, linear/nonlinear dye
bias, beta and M-values (MET-010..012), the 21 methylation control metrics (MET-006,
bs-conv red-bg set option), per-sample ctrl/cgs tables, summary tsv/xlsx/html with the
detection histogram and a beta PCA, pcs.tsv.gz, controls_raw/controls_qc.tsv; methylation
datasets load in qc report (MET-014); qc controls --array-type methylation; minfi-style
Sentrix_ID/Sentrix_Position sheet keys. Vendor EPIC manifests carry a 5th field on some
control lines: >= 4 fields accepted. Runs on the public minfiDataEPIC set (3 x NA12878).
2026-09-29 00:13:22 +01:00
kemal
c1699b4465
P6: gtc-to-vcf (VCF/BCF + indexes), harmonisation, exports (plink, bedgraph, finalreport)
...
- formats/fasta: plain/BGZF FASTA, .fai/.gzi built identical to htslib, index check, build detection,
contig aliases, mmap genome cache (IN-060..064)
- formats/bgzf, formats/bcf: pure-Python BGZF, tabix .tbi, BCF 2.2 and CSI (OUT-010, OUT-038, OUT-040)
- harmonize/sites: plus-strand mapping with audit rows, indel resolution from SourceSeq (port of GTCtoVCF,
Apache-2.0) and left normalisation (OUT-014..021, OUT-030, OUT-042)
- harmonize/combine: OUT-031 genotype combination (ploidy-aware, Infinium I/II signal classes, OUT-037)
- pipeline/vcf: `simurg gtc-to-vcf` (per-sample VCF.gz+tbi, --merge cohort BCF+csi, audit parquet)
- pipeline/export: `simurg export bedgraph|finalreport|plink|locus-summary` (OUT-043, OUT-050..052, OUT-071)
- validation: 36 GSA samples vs gtc2vcf: 23.66 M identical calls, 0 unexplained differences; PLINK identical
to plink2 on 637,397 variants; bcftools validates VCF/BCF and our indexes answer region queries correctly
- fetch_data.sh p6 (Ensembl GRCh38 with checksum), make_oracle_vcf.sh, validate_p6_vcf.py; K-022/K-023
2026-09-28 12:17:53 +01:00
kemal
a8caa26aa4
P5a: QC — control metrics, thresholds, status, contamination, relatedness, locus summary
...
- qc/controls: canonical control names (QC-002), bkg/offset (QC-003), 13 genotyping metrics (QC-004/005),
missing/zero-divisor handling (QC-007/008); controls_raw.tsv / controls_qc.tsv (QC-009/011) incl. compat CSVs
- qc/thresholds + profiles: YAML schema, validation codes, precedence, shipped profiles, auto (QC-020..025)
- qc/status: functional PASS/FAIL/NA, control CLEAR/FLAGGED, flags (QC-030..033)
- qc/contamination: BAF regression; bit-compatible with gtc2vcf BAFregress (36/36, K-017), exact mode too
- qc/relatedness: KING-robust, duplicates, Mendelian errors, pedigree from sheet (IN-018)
- qc/locus_summary: OUT-072, bit-identical to BeadArrayFiles except ChiTest (K-018)
- qc/suggest: suggest-thresholds (QC-026)
- pipeline/qc: disk-backed cohort store; CLI qc controls|samples|contamination|relatedness|loci|suggest-thresholds
- genotype: control tables (OUT-070), het_rate, QC columns, cohort QC
- tests: unit + oracle (BAFregress plugin, BeadArrayFiles locus summary)
2026-09-28 08:35:10 +01:00
kemal
1f7b6d10c5
P4 (WIP): GenCall-equivalent calling, BAF/LRR, sex, GTC v5, simurg genotype
...
- genotype/clusters.py: EGT aligned to manifest order (float32 mean R)
- genotype/gencall.py: flair, six-case likelihood, priors/posteriors,
fuzzy sub-scores and score map, vectorised with the reference's
float/double evaluation order and libm transcendentals
- genotype/baf_lrr.py, sex.py (rules v1/v2), sample.py (IDAT -> Gtc)
- pipeline/inputs.py shared by ingest and genotype
- pipeline/genotype.py + CLI: GTC per sample, sample_summary.tsv/json
(OUT-060..063), run manifest
- GTCs from IDATs are byte-identical to the oracle for 36 GSA samples x
2 normalization versions (NaN sign canonicalised to the x86 pattern)
2026-09-28 00:07:37 +01:00
kemal
87f9b3499c
P2 (WIP): raw X/Y assembly, canonical Parquet model, simurg ingest
...
- signal/raw_xy.py: vectorised Infinium I/II raw X/Y and control assembly
(plan §7.1) with a per-chip address-map cache; IN-027 coverage gate
- core/schemas.py: Arrow schemas (loci, sample_intensity, controls),
deterministic atomic Parquet writing
- pipeline/ingest.py + CLI: discovery, sheet, IN-010 naming, overwrite
check before any work, spawn-based workers, per-sample isolation,
--resume by input hashes, run manifest, exit codes 0/3/4
- matches gtc2vcf raw X/Y and controls exactly on 18 GSA samples;
outputs byte-identical across thread counts and layouts
2026-09-27 20:54:06 +01:00
kemal
8cd092460f
P1: file parsers, sample sheet, discovery, CLI inspect
...
Readers for IDAT v3, BPM (locus entries v4-v8), CSV manifest, EGT
(data blocks 5-9) and GTC v3-v5 (plus a layout-preserving GTC writer);
sample-sheet validation (IN-001..IN-014), IDAT discovery and pairing
(IN-020..IN-022), message catalogue and exit codes (CLI-014/016),
JSON Lines logging (CLI-007/008), atomic writes (CLI-011), run manifest,
and `simurg inspect`.
Validation (docs/validation/P1.md): GTC, BPM and EGT readers agree
exactly with Illumina BeadArrayFiles on GSA v4 / GDA v1 files; GTCs
round-trip byte-identically; BPM and CSV manifests are consistent.
2026-09-27 16:56:39 +01:00
kemal