kemal
3c33ae5cb1
- compare/core: EXACT/NEAR/DISCREPANCY/UNKNOWN with the plan §16.3 tolerances (abs, float32 ULP), NaN = NaN, locus classes (assay, variant type, chromosome group, intensity-only); fields absent on both sides are not applicable - compare/records: GTC vs GTC (per-locus arrays, transforms per bin, controls, sample statistics; metadata listed, not counted) and VCF vs VCF (records keyed incl. ID; SNV/MNV/indel, merged, multi-allelic classes) - simurg compare A B (files or folders paired by name): summary.json, discrepancies.tsv.gz (never truncated), self-contained dashboard.html; --tolerance, --fail-on-discrepancy (exit 3) - scripts/nightly.py + install_nightly.sh (crontab 02:30): unit tests, GSA vs gtc2vcf (GTC + VCF), VCF regression vs golden, Omni2.5 subset vs vendor GTCs; first run PASS on every step - tests/unit/test_compare.py; docs/validation/P7.md, README
47 lines
3.3 KiB
Markdown
47 lines
3.3 KiB
Markdown
# SimurgArray
|
||
|
||
Personal research toolkit for Illumina Infinium genotyping BeadChip data
|
||
(IDAT → normalization → genotype calling → BAF/LRR → QC → VCF/PLINK; later CN/LOH, PGx, cluster training).
|
||
|
||
- **SIMURGARRAY_MASTER_PLAN.md** — architecture, algorithms, roadmap
|
||
- **SIMURGARRAY_FUNCTIONAL_SPEC.md** — 416 itemised requirements (CLI, inputs, outputs, QC, CN, PGx, methylation, edge cases)
|
||
- `docs/validation/` — per-phase validation reports · `docs/known_unknowns.md` — findings log · `docs/research/` — background
|
||
|
||
## Status
|
||
|
||
Phases **P0 (setup)**, **P1 (file parsers)**, **P2 (canonical model, raw X/Y)**, **P3 (normalization)** and **P4 (genotype
|
||
calling, GTC output)** are complete — see `docs/validation/`. Each is validated on 2141 vendor GTCs of the 1000 Genomes Omni2.5 set
|
||
(normalization bit-identical; genotypes/scores identical in 2138/2141 samples).
|
||
Phase **P5 (QC)** is complete: control-probe metrics, thresholds/profiles, two-axis status, BAF-regression contamination,
|
||
KING-robust relatedness, Mendelian errors, locus summary and a self-contained HTML report (`simurg qc report`). On 2141
|
||
1000 Genomes samples all 717 pedigree parent–offspring pairs are recovered and trio Mendelian error rates are ~0.03 %
|
||
(`docs/validation/P5.md`).
|
||
Phase **P6 (outputs)** is complete: `simurg gtc-to-vcf` (plus-strand harmonisation with audit, merged multi-probe
|
||
sites, per-sample VCF.gz + tabix or a cohort BCF + CSI, optional INFO/intensity fields, SNV+indel merging, MNV
|
||
auxiliary loci) and `simurg export plink|bedgraph|finalreport|locus-summary` (PLINK 1 `.bed` and PLINK 2 `.pgen`).
|
||
On the GSA demo, genotypes match gtc2vcf and GTCtoVCF with zero unexplained differences and PLINK files match plink2
|
||
(`docs/validation/P6.md`).
|
||
Phase **P7 (differential testing)** is complete: `simurg compare` (EXACT/NEAR/DISCREPANCY/UNKNOWN per field and
|
||
locus class, every discrepancy listed, HTML dashboard) and a nightly run on the Pi (GSA vs gtc2vcf, Omni2.5 subset vs
|
||
vendor GTCs, VCF regression) — `docs/validation/P7.md`.
|
||
Available commands: `simurg version`, `simurg inspect <files…> [--json]`, `simurg ingest`, `simurg normalize`, `simurg genotype`,
|
||
`simurg qc controls|samples|contamination|relatedness|loci|report|suggest-thresholds`, `simurg gtc-to-vcf`,
|
||
`simurg export plink|bedgraph|finalreport|locus-summary`, `simurg compare`.
|
||
|
||
## Quick start
|
||
|
||
```bash
|
||
uv venv && uv pip install -e ".[dev]"
|
||
uv run pytest tests/unit # no downloads needed
|
||
bash scripts/bootstrap_refs.sh # reference open-source repos → refs/ (oracles)
|
||
bash scripts/fetch_data.sh p0 # ~1.4 GB of public test data → data/
|
||
bash scripts/make_oracle_gtcs.sh # builds gtc2vcf, writes oracle GTCs
|
||
bash scripts/make_oracle_qc.sh # locus-summary oracle (BeadArrayFiles, separate venv)
|
||
bash scripts/fetch_data.sh p6 # GRCh38 FASTA (~0.9 GB download, 3.1 GB unpacked)
|
||
bash scripts/make_oracle_vcf.sh # gtc2vcf oracle VCF for the P6 tests
|
||
uv run pytest tests/oracle # real-data acceptance tests (add -m "not slow" to skip the long ones)
|
||
uv run simurg inspect data/p0/gsa_v4_pgx/idats_Demo_36/idats_Demo_36/207895980019/207895980019_R03C01_Grn.idat
|
||
```
|
||
|
||
Vendor data files are downloaded for personal research use only and are never committed. See `THIRD_PARTY_NOTICES.md`.
|