kemal/SimurgArray
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
38 Commits 1 Branch 3 Tags
bc93e003183cfefa509415e1615f7a47aebb8f3f
Commit Graph
7 Commits
Author SHA1 Message Date
kemalandClaude Opus 5.5 bc93e00318 CLI: a closed stdout pipe exits 141 silently (was E_INTERNAL); example avoids head under pipefail
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
Found by the P10 fresh-VM run: `simurg inspect … | head` raised BrokenPipeError, reported as
an internal error, and stopped examples/gsa_demo.sh (set -o pipefail). run_guarded now flushes
stdout inside the guard and maps BrokenPipeError to ExitCode.SIGPIPE (141) without a message.
p10.sh: SKIP_DOCKER/SKIP_CONDA, clearer digest report; p10t.sh runs build+pip+tutorial only.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-28 23:14:47 +01:00
kemalandClaude Opus 5.5 b56fdebaaf P6: gtc-to-vcf (VCF/BCF + indexes), harmonisation, exports (plink, bedgraph, finalreport) 
- formats/fasta: plain/BGZF FASTA, .fai/.gzi built identical to htslib, index check, build detection,
  contig aliases, mmap genome cache (IN-060..064)
- formats/bgzf, formats/bcf: pure-Python BGZF, tabix .tbi, BCF 2.2 and CSI (OUT-010, OUT-038, OUT-040)
- harmonize/sites: plus-strand mapping with audit rows, indel resolution from SourceSeq (port of GTCtoVCF,
  Apache-2.0) and left normalisation (OUT-014..021, OUT-030, OUT-042)
- harmonize/combine: OUT-031 genotype combination (ploidy-aware, Infinium I/II signal classes, OUT-037)
- pipeline/vcf: `simurg gtc-to-vcf` (per-sample VCF.gz+tbi, --merge cohort BCF+csi, audit parquet)
- pipeline/export: `simurg export bedgraph|finalreport|plink|locus-summary` (OUT-043, OUT-050..052, OUT-071)
- validation: 36 GSA samples vs gtc2vcf: 23.66 M identical calls, 0 unexplained differences; PLINK identical
  to plink2 on 637,397 variants; bcftools validates VCF/BCF and our indexes answer region queries correctly
- fetch_data.sh p6 (Ensembl GRCh38 with checksum), make_oracle_vcf.sh, validate_p6_vcf.py; K-022/K-023

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-28 12:17:53 +01:00
kemalandClaude Opus 5.5 0c004f6398 P5a: QC — control metrics, thresholds, status, contamination, relatedness, locus summary
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 11s
Details 
- qc/controls: canonical control names (QC-002), bkg/offset (QC-003), 13 genotyping metrics (QC-004/005),
  missing/zero-divisor handling (QC-007/008); controls_raw.tsv / controls_qc.tsv (QC-009/011) incl. compat CSVs
- qc/thresholds + profiles: YAML schema, validation codes, precedence, shipped profiles, auto (QC-020..025)
- qc/status: functional PASS/FAIL/NA, control CLEAR/FLAGGED, flags (QC-030..033)
- qc/contamination: BAF regression; bit-compatible with gtc2vcf BAFregress (36/36, K-017), exact mode too
- qc/relatedness: KING-robust, duplicates, Mendelian errors, pedigree from sheet (IN-018)
- qc/locus_summary: OUT-072, bit-identical to BeadArrayFiles except ChiTest (K-018)
- qc/suggest: suggest-thresholds (QC-026)
- pipeline/qc: disk-backed cohort store; CLI qc controls|samples|contamination|relatedness|loci|suggest-thresholds
- genotype: control tables (OUT-070), het_rate, QC columns, cohort QC
- tests: unit + oracle (BAFregress plugin, BeadArrayFiles locus summary)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-28 08:35:10 +01:00
kemalandClaude Opus 5.5 febdd121ef P4 (WIP): GenCall-equivalent calling, BAF/LRR, sex, GTC v5, simurg genotype
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
- genotype/clusters.py: EGT aligned to manifest order (float32 mean R)
- genotype/gencall.py: flair, six-case likelihood, priors/posteriors,
  fuzzy sub-scores and score map, vectorised with the reference's
  float/double evaluation order and libm transcendentals
- genotype/baf_lrr.py, sex.py (rules v1/v2), sample.py (IDAT -> Gtc)
- pipeline/inputs.py shared by ingest and genotype
- pipeline/genotype.py + CLI: GTC per sample, sample_summary.tsv/json
  (OUT-060..063), run manifest
- GTCs from IDATs are byte-identical to the oracle for 36 GSA samples x
  2 normalization versions (NaN sign canonicalised to the x86 pattern)

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-28 00:07:37 +01:00
kemalandClaude Opus 5.5 314fdc40d1 P3: normalization fit (1.2.0 and 1.1.2), apply, polar, simurg normalize
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
- signal/normalization/numerics.py: vendor percentile, trimmean, MAD sigma
  (old/new), bisquare robust fits, float32/float64 typing and sequential
  sums reproduced exactly (ported from gtc2vcf, MIT)
- signal/normalization/fit.py: outlier removal, offset, rotation, shear,
  scale, nn12/rr12, single-channel model for small Infinium I bins,
  exact and binned nearest-neighbour sweeps; vendor quirks marked QUIRK
- signal/normalization/apply.py: affine transform and theta/R
- pipeline/normalize.py + CLI `simurg normalize`
- oracle GTCs rebuilt with -ffp-contract=off (aarch64 gcc fuses a*b+c by
  default, shifting the last bits); the old FMA oracle differed by <=4 ULP

Result: transforms bit-identical to the oracle for all 36 GSA samples x
48 bins, for both normalization versions (3456/3456 bins, 0 ULP).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 22:51:06 +01:00
kemalandClaude Opus 5.5 a4faead6d7 P2 (WIP): raw X/Y assembly, canonical Parquet model, simurg ingest
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
- signal/raw_xy.py: vectorised Infinium I/II raw X/Y and control assembly
  (plan §7.1) with a per-chip address-map cache; IN-027 coverage gate
- core/schemas.py: Arrow schemas (loci, sample_intensity, controls),
  deterministic atomic Parquet writing
- pipeline/ingest.py + CLI: discovery, sheet, IN-010 naming, overwrite
  check before any work, spawn-based workers, per-sample isolation,
  --resume by input hashes, run manifest, exit codes 0/3/4
- matches gtc2vcf raw X/Y and controls exactly on 18 GSA samples;
  outputs byte-identical across thread counts and layouts

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 20:54:06 +01:00
kemalandClaude Opus 5.5 4357d3deef P1: file parsers, sample sheet, discovery, CLI inspect
ci / test (3.12) (push) Failing after 11s
Details
ci / test (3.13) (push) Failing after 12s
Details 
Readers for IDAT v3, BPM (locus entries v4-v8), CSV manifest, EGT
(data blocks 5-9) and GTC v3-v5 (plus a layout-preserving GTC writer);
sample-sheet validation (IN-001..IN-014), IDAT discovery and pairing
(IN-020..IN-022), message catalogue and exit codes (CLI-014/016),
JSON Lines logging (CLI-007/008), atomic writes (CLI-011), run manifest,
and `simurg inspect`.

Validation (docs/validation/P1.md): GTC, BPM and EGT readers agree
exactly with Illumina BeadArrayFiles on GSA v4 / GDA v1 files; GTCs
round-trip byte-identically; BPM and CSV manifests are consistent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 16:56:39 +01:00