kemalandClaude Opus 5.5
d0f8c3a144
360 GSA arrays (120 1000 Genomes individuals x 3 replicates): concordance 0.9934 vs NYGC 30x (0.9958 on sites in both sets), 0.9985 vs GIAB HG001 v4.2.1 (non-ref F1 0.997), replicate concordance 0.9966; allele-swap sites 0.016 %; 358/360 replicate pairs detected as duplicates. Plan §16.5 metrics in docs/validation/P8.md (K-024). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
3.6 KiB
SimurgArray
Personal research toolkit for Illumina Infinium genotyping BeadChip data (IDAT → normalization → genotype calling → BAF/LRR → QC → VCF/PLINK; later CN/LOH, PGx, cluster training).
- SIMURGARRAY_MASTER_PLAN.md — architecture, algorithms, roadmap
- SIMURGARRAY_FUNCTIONAL_SPEC.md — 416 itemised requirements (CLI, inputs, outputs, QC, CN, PGx, methylation, edge cases)
docs/validation/— per-phase validation reports ·docs/known_unknowns.md— findings log ·docs/research/— background
Status
Phases P0 (setup), P1 (file parsers), P2 (canonical model, raw X/Y), P3 (normalization) and P4 (genotype
calling, GTC output) are complete — see docs/validation/. Each is validated on 2141 vendor GTCs of the 1000 Genomes Omni2.5 set
(normalization bit-identical; genotypes/scores identical in 2138/2141 samples).
Phase P5 (QC) is complete: control-probe metrics, thresholds/profiles, two-axis status, BAF-regression contamination,
KING-robust relatedness, Mendelian errors, locus summary and a self-contained HTML report (simurg qc report). On 2141
1000 Genomes samples all 717 pedigree parent–offspring pairs are recovered and trio Mendelian error rates are ~0.03 %
(docs/validation/P5.md).
Phase P6 (outputs) is complete: simurg gtc-to-vcf (plus-strand harmonisation with audit, merged multi-probe
sites, per-sample VCF.gz + tabix or a cohort BCF + CSI, optional INFO/intensity fields, SNV+indel merging, MNV
auxiliary loci) and simurg export plink|bedgraph|finalreport|locus-summary (PLINK 1 .bed and PLINK 2 .pgen).
On the GSA demo, genotypes match gtc2vcf and GTCtoVCF with zero unexplained differences and PLINK files match plink2
(docs/validation/P6.md).
Phase P7 (differential testing) is complete: simurg compare (EXACT/NEAR/DISCREPANCY/UNKNOWN per field and
locus class, every discrepancy listed, HTML dashboard) and a nightly run on the Pi (GSA vs gtc2vcf, Omni2.5 subset vs
vendor GTCs, VCF regression) — docs/validation/P7.md.
Phase P8 (scientific validation) is complete: on GEO GSE165845 (360 GSA arrays, 120 1000 Genomes individuals ×
3 replicates) genotype concordance is 0.9934 vs NYGC 30x WGS, 0.9985 vs GIAB HG001 v4.2.1 (non-ref F1 0.997), and
0.9966 between replicates — docs/validation/P8.md.
Available commands: simurg version, simurg inspect <files…> [--json], simurg ingest, simurg normalize, simurg genotype,
simurg qc controls|samples|contamination|relatedness|loci|report|suggest-thresholds, simurg gtc-to-vcf,
simurg export plink|bedgraph|finalreport|locus-summary, simurg compare.
Quick start
uv venv && uv pip install -e ".[dev]"
uv run pytest tests/unit # no downloads needed
bash scripts/bootstrap_refs.sh # reference open-source repos → refs/ (oracles)
bash scripts/fetch_data.sh p0 # ~1.4 GB of public test data → data/
bash scripts/make_oracle_gtcs.sh # builds gtc2vcf, writes oracle GTCs
bash scripts/make_oracle_qc.sh # locus-summary oracle (BeadArrayFiles, separate venv)
bash scripts/fetch_data.sh p6 # GRCh38 FASTA (~0.9 GB download, 3.1 GB unpacked)
bash scripts/make_oracle_vcf.sh # gtc2vcf oracle VCF for the P6 tests
uv run pytest tests/oracle # real-data acceptance tests (add -m "not slow" to skip the long ones)
uv run simurg inspect data/p0/gsa_v4_pgx/idats_Demo_36/idats_Demo_36/207895980019/207895980019_R03C01_Grn.idat
Vendor data files are downloaded for personal research use only and are never committed. See THIRD_PARTY_NOTICES.md.