kemalandClaude Opus 5.5
cd86681742
- compare/core: EXACT/NEAR/DISCREPANCY/UNKNOWN with the plan §16.3 tolerances (abs, float32 ULP), NaN = NaN, locus classes (assay, variant type, chromosome group, intensity-only); fields absent on both sides are not applicable - compare/records: GTC vs GTC (per-locus arrays, transforms per bin, controls, sample statistics; metadata listed, not counted) and VCF vs VCF (records keyed incl. ID; SNV/MNV/indel, merged, multi-allelic classes) - simurg compare A B (files or folders paired by name): summary.json, discrepancies.tsv.gz (never truncated), self-contained dashboard.html; --tolerance, --fail-on-discrepancy (exit 3) - scripts/nightly.py + install_nightly.sh (crontab 02:30): unit tests, GSA vs gtc2vcf (GTC + VCF), VCF regression vs golden, Omni2.5 subset vs vendor GTCs; first run PASS on every step - tests/unit/test_compare.py; docs/validation/P7.md, README Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
39 lines
2.9 KiB
Markdown
39 lines
2.9 KiB
Markdown
# Phase P7 validation — differential-testing harness
|
|
|
|
Date: 2026-09-28. All work on the Raspberry Pi (no cloud cost).
|
|
|
|
## Delivered
|
|
|
|
| Item | Module |
|
|
|---|---|
|
|
| Tolerance categories EXACT / NEAR / DISCREPANCY / UNKNOWN with the plan §16.3 tolerances; float32 ULP distance; NaN = NaN regardless of sign (K-014) | `compare/core.py` |
|
|
| Locus classes (assay, variant type, chromosome group incl. PAR/MT/unmapped, intensity-only) | `compare/core.py` |
|
|
| GTC-vs-GTC (every per-locus array, transforms per bin, controls, sample statistics; free-text metadata reported but not counted) and VCF-vs-VCF (per record and FORMAT field, records keyed by CHROM/POS/REF/ALT/ID, classes SNV/MNV/indel, merged sites, multi-allelic) | `compare/records.py` |
|
|
| `simurg compare A B` (files or folders paired by name; `--tolerance FIELD=…`, `--fail-on-discrepancy`) → `summary.json`, `discrepancies.tsv.gz` (every discrepancy and unmatched record, never truncated), self-contained `dashboard.html` | `pipeline/compare.py` |
|
|
| Nightly run: unit tests, GSA genotype vs gtc2vcf oracle, GSA VCF vs gtc2vcf (semantic) and vs golden (regression), Omni2.5 subset vs vendor GTCs; per-night page + history; 14 nights kept | `scripts/nightly.py`, `scripts/install_nightly.sh` (crontab, 02:30) |
|
|
|
|
## Acceptance (plan §18 P7: nightly run on P0 + Omni subset produces report)
|
|
|
|
First complete nightly run on the Pi (2026-09-28, 17 min):
|
|
|
|
| Step | Compared | Result |
|
|
|---|---|---|
|
|
| unit | 206 unit tests | PASS |
|
|
| gsa_gtc | 36 GSA GTCs from IDATs vs gtc2vcf oracle GTCs, 20 fields, per locus class | **PASS**: 0 discrepancies, 0 unknown (free-text metadata listed separately) |
|
|
| gsa_vcf | 36 VCFs vs gtc2vcf (semantic, `validate_p6_vcf.py`) | **PASS**: 0 unexplained |
|
|
| gsa_vcf regression | 36 VCFs vs the golden set of the first run: 23 640 768 records | all records present, GT/GS/BAF/LRR EXACT (see note on BAF/LRR of merged sites) |
|
|
| omni_gtc | 6 Omni2.5 samples (`--preset autocall-1.6.2`) vs the vendor's AutoCall 1.6.2.2 GTCs, 18 fields | **PASS**: 0 discrepancies, 0 unknown |
|
|
|
|
Installed as a crontab entry (every night 02:30, `flock` against overlapping runs; `bash scripts/install_nightly.sh
|
|
--remove` removes it). History page: `data/nightly/index.html`.
|
|
|
|
## Notes
|
|
|
|
* The harness found its own bug while being built: VCF records keyed without ID let duplicate designs overwrite
|
|
each other in `--unsquash-duplicates` output; keys now include the ID and repeated keys are counted, never merged.
|
|
* The first regression run reported 51 048 UNKNOWN values: BAF/LRR of multi-allelic merged records, which carry
|
|
`GT:GS` only on *both* sides. A field absent on both sides is now "not applicable" and not counted (GTC and VCF
|
|
alike); a field absent on one side stays UNKNOWN.
|
|
* Resource use on the Pi: one Omni2.5 sample genotypes in 93 s at 3.4 GB peak RSS (single process); the GSA VCF
|
|
step memory-maps the 3.1 GB genome cache, so its address-space cap is 12 GB (page cache, not RAM).
|