kemalandClaude Opus 5.5
cd86681742
- compare/core: EXACT/NEAR/DISCREPANCY/UNKNOWN with the plan §16.3 tolerances (abs, float32 ULP), NaN = NaN, locus classes (assay, variant type, chromosome group, intensity-only); fields absent on both sides are not applicable - compare/records: GTC vs GTC (per-locus arrays, transforms per bin, controls, sample statistics; metadata listed, not counted) and VCF vs VCF (records keyed incl. ID; SNV/MNV/indel, merged, multi-allelic classes) - simurg compare A B (files or folders paired by name): summary.json, discrepancies.tsv.gz (never truncated), self-contained dashboard.html; --tolerance, --fail-on-discrepancy (exit 3) - scripts/nightly.py + install_nightly.sh (crontab 02:30): unit tests, GSA vs gtc2vcf (GTC + VCF), VCF regression vs golden, Omni2.5 subset vs vendor GTCs; first run PASS on every step - tests/unit/test_compare.py; docs/validation/P7.md, README Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
2.9 KiB
2.9 KiB
Phase P7 validation — differential-testing harness
Date: 2026-09-28. All work on the Raspberry Pi (no cloud cost).
Delivered
| Item | Module |
|---|---|
| Tolerance categories EXACT / NEAR / DISCREPANCY / UNKNOWN with the plan §16.3 tolerances; float32 ULP distance; NaN = NaN regardless of sign (K-014) | compare/core.py |
| Locus classes (assay, variant type, chromosome group incl. PAR/MT/unmapped, intensity-only) | compare/core.py |
| GTC-vs-GTC (every per-locus array, transforms per bin, controls, sample statistics; free-text metadata reported but not counted) and VCF-vs-VCF (per record and FORMAT field, records keyed by CHROM/POS/REF/ALT/ID, classes SNV/MNV/indel, merged sites, multi-allelic) | compare/records.py |
simurg compare A B (files or folders paired by name; --tolerance FIELD=…, --fail-on-discrepancy) → summary.json, discrepancies.tsv.gz (every discrepancy and unmatched record, never truncated), self-contained dashboard.html |
pipeline/compare.py |
| Nightly run: unit tests, GSA genotype vs gtc2vcf oracle, GSA VCF vs gtc2vcf (semantic) and vs golden (regression), Omni2.5 subset vs vendor GTCs; per-night page + history; 14 nights kept | scripts/nightly.py, scripts/install_nightly.sh (crontab, 02:30) |
Acceptance (plan §18 P7: nightly run on P0 + Omni subset produces report)
First complete nightly run on the Pi (2026-09-28, 17 min):
| Step | Compared | Result |
|---|---|---|
| unit | 206 unit tests | PASS |
| gsa_gtc | 36 GSA GTCs from IDATs vs gtc2vcf oracle GTCs, 20 fields, per locus class | PASS: 0 discrepancies, 0 unknown (free-text metadata listed separately) |
| gsa_vcf | 36 VCFs vs gtc2vcf (semantic, validate_p6_vcf.py) |
PASS: 0 unexplained |
| gsa_vcf regression | 36 VCFs vs the golden set of the first run: 23 640 768 records | all records present, GT/GS/BAF/LRR EXACT (see note on BAF/LRR of merged sites) |
| omni_gtc | 6 Omni2.5 samples (--preset autocall-1.6.2) vs the vendor's AutoCall 1.6.2.2 GTCs, 18 fields |
PASS: 0 discrepancies, 0 unknown |
Installed as a crontab entry (every night 02:30, flock against overlapping runs; bash scripts/install_nightly.sh --remove removes it). History page: data/nightly/index.html.
Notes
- The harness found its own bug while being built: VCF records keyed without ID let duplicate designs overwrite
each other in
--unsquash-duplicatesoutput; keys now include the ID and repeated keys are counted, never merged. - The first regression run reported 51 048 UNKNOWN values: BAF/LRR of multi-allelic merged records, which carry
GT:GSonly on both sides. A field absent on both sides is now "not applicable" and not counted (GTC and VCF alike); a field absent on one side stays UNKNOWN. - Resource use on the Pi: one Omni2.5 sample genotypes in 93 s at 3.4 GB peak RSS (single process); the GSA VCF step memory-maps the 3.1 GB genome cache, so its address-space cap is 12 GB (page cache, not RAM).