kemalandClaude Opus 5.5
b56fdebaaf
- formats/fasta: plain/BGZF FASTA, .fai/.gzi built identical to htslib, index check, build detection, contig aliases, mmap genome cache (IN-060..064) - formats/bgzf, formats/bcf: pure-Python BGZF, tabix .tbi, BCF 2.2 and CSI (OUT-010, OUT-038, OUT-040) - harmonize/sites: plus-strand mapping with audit rows, indel resolution from SourceSeq (port of GTCtoVCF, Apache-2.0) and left normalisation (OUT-014..021, OUT-030, OUT-042) - harmonize/combine: OUT-031 genotype combination (ploidy-aware, Infinium I/II signal classes, OUT-037) - pipeline/vcf: `simurg gtc-to-vcf` (per-sample VCF.gz+tbi, --merge cohort BCF+csi, audit parquet) - pipeline/export: `simurg export bedgraph|finalreport|plink|locus-summary` (OUT-043, OUT-050..052, OUT-071) - validation: 36 GSA samples vs gtc2vcf: 23.66 M identical calls, 0 unexplained differences; PLINK identical to plink2 on 637,397 variants; bcftools validates VCF/BCF and our indexes answer region queries correctly - fetch_data.sh p6 (Ensembl GRCh38 with checksum), make_oracle_vcf.sh, validate_p6_vcf.py; K-022/K-023 Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
64 lines
3.8 KiB
Markdown
64 lines
3.8 KiB
Markdown
# Phase P6 validation — outputs (VCF/BCF, PLINK, bedGraph, final report)
|
||
|
||
Date: 2026-09-28. All work on the Raspberry Pi (no cloud cost). Data: the 36 GSA v4 PGx oracle GTCs (P4, byte-identical
|
||
to gtc2vcf's), the GSA BPM + CSV manifest, and the Ensembl GRCh38 primary assembly (release 110, `fetch_data.sh p6`).
|
||
|
||
## Delivered
|
||
|
||
| Item | Module |
|
||
|---|---|
|
||
| FASTA access: plain/BGZF, `.fai`/`.gzi` read and built (identical to htslib's), index check, build detection, contig aliases, mmap genome cache | `formats/fasta.py` |
|
||
| BGZF writer, tabix `.tbi`, BCF 2.2 encoder, CSI index (pure Python) | `formats/bgzf.py`, `formats/bcf.py` |
|
||
| Plus-strand mapping with audit row per locus; indel resolution from `SourceSeq` (context score), left normalisation | `harmonize/sites.py` |
|
||
| Site grouping and the OUT-031 genotype-combination algorithm (ploidy-aware, Infinium I/II signal classes) | `harmonize/combine.py` |
|
||
| `simurg gtc-to-vcf` (per-sample VCF.gz + .tbi, `--merge` cohort BCF + .csi, audit Parquet) | `pipeline/vcf.py` |
|
||
| `simurg export bedgraph | finalreport | plink | locus-summary` | `pipeline/export.py` |
|
||
|
||
## Acceptance (plan §18 P6)
|
||
|
||
### bcftools validation and indexes
|
||
|
||
Every VCF passes `bcftools view`; `bcftools index --nrecords` on our `.tbi` equals the record count (635 796); region
|
||
queries through our index (`1:1-2M`, `X:150-156M`, `MT`, `Y`, `22:30.0-30.1M`) return exactly the records a full scan
|
||
finds. The cohort BCF reads back identically to the per-sample VCFs (635 796 records × 3 samples, 0 differences in
|
||
GT/GS/BAF/LRR) and its CSI index answers region queries.
|
||
|
||
### Genotypes vs gtc2vcf (MIT), 36 samples, one record per probe (`--unsquash-duplicates`)
|
||
|
||
Both sides left-aligned with `bcftools norm -f`, compared per probe and sample as allele sets.
|
||
|
||
| Category | Count (probe × sample) | Explanation |
|
||
|---|---|---|
|
||
| identical calls | 23 656 010 | |
|
||
| identical no-calls | 102 077 | |
|
||
| our merged multi-allelic site differs from the single-probe call | 2 414 | e.g. an Infinium II [G/T] probe calls GG while another probe at the site shows G/C: C shares G's dye, so the probe cannot see it — our site genotype is consistent with every probe |
|
||
| our merged site missing, probe called | 3 945 | the probes at the site contradict each other (`W_MERGE_INCONSISTENT`) or leave several genotypes (`W_MERGE_AMBIGUOUS`) |
|
||
| our merged site called, probe no-call | 65 | other probes at the site resolve it |
|
||
| het in a haploid context → missing (OUT-018) | 14 | male X non-PAR and MT heterozygous calls |
|
||
| female Y → missing (OUT-018) | 11 | |
|
||
| indels gtc2vcf leaves unresolved (`REF=I/D`) that we resolve | 144 probes | checked against GTCtoVCF below |
|
||
| **unexplained differences** | **0** | |
|
||
|
||
Locus sets: gtc2vcf writes 22 400 intensity-only loci that SimurgArray skips by design (OUT-019), and 49 indels it
|
||
cannot resolve either (`W_SKIP_INDEL_REF` for us). Nothing we write is missing from gtc2vcf.
|
||
|
||
### Genotypes vs GTCtoVCF (Apache-2.0), 36 samples, merged sites
|
||
|
||
(filled in below)
|
||
|
||
### PLINK vs plink2 (GPL, subprocess oracle, built from source)
|
||
|
||
`simurg export plink` and `plink2 --bcf <bcftools norm -m- of our BCF> --make-bed --split-par hg38` agree on all
|
||
637 397 variants × 3 samples (bed codes identical; the 1 601 extra variants are split multi-allelic sites).
|
||
|
||
### BAF orientation, bedGraph
|
||
|
||
* EDGE-017: at every called homozygous single-probe site where the probe's A allele is ALT, VCF BAF = 1 − GTC BAF
|
||
(float32), otherwise unchanged (> 500 sites of each kind checked per sample).
|
||
* EDGE-023: raw LRR bedGraph values equal the GTC values after `%.6g` formatting for every single-probe position.
|
||
|
||
### Not yet implemented (listed in known_unknowns K-023)
|
||
|
||
`--auxiliary-loci` (OUT-035), `--info-fields` and `--extra-format` (OUT-012), `--merge-snv-indel` (OUT-021),
|
||
PLINK 2 `.pgen` output.
|