0 Fork 0

Files

T

kemal c1699b4465 P6: gtc-to-vcf (VCF/BCF + indexes), harmonisation, exports (plink, bedgraph, finalreport)

- formats/fasta: plain/BGZF FASTA, .fai/.gzi built identical to htslib, index check, build detection,
  contig aliases, mmap genome cache (IN-060..064)
- formats/bgzf, formats/bcf: pure-Python BGZF, tabix .tbi, BCF 2.2 and CSI (OUT-010, OUT-038, OUT-040)
- harmonize/sites: plus-strand mapping with audit rows, indel resolution from SourceSeq (port of GTCtoVCF,
  Apache-2.0) and left normalisation (OUT-014..021, OUT-030, OUT-042)
- harmonize/combine: OUT-031 genotype combination (ploidy-aware, Infinium I/II signal classes, OUT-037)
- pipeline/vcf: `simurg gtc-to-vcf` (per-sample VCF.gz+tbi, --merge cohort BCF+csi, audit parquet)
- pipeline/export: `simurg export bedgraph|finalreport|plink|locus-summary` (OUT-043, OUT-050..052, OUT-071)
- validation: 36 GSA samples vs gtc2vcf: 23.66 M identical calls, 0 unexplained differences; PLINK identical
  to plink2 on 637,397 variants; bcftools validates VCF/BCF and our indexes answer region queries correctly
- fetch_data.sh p6 (Ensembl GRCh38 with checksum), make_oracle_vcf.sh, validate_p6_vcf.py; K-022/K-023

2026-09-28 12:17:53 +01:00

3.8 KiB

Raw Blame History

Phase P6 validation — outputs (VCF/BCF, PLINK, bedGraph, final report)

Date: 2026-09-28. All work on the Raspberry Pi (no cloud cost). Data: the 36 GSA v4 PGx oracle GTCs (P4, byte-identical to gtc2vcf's), the GSA BPM + CSV manifest, and the Ensembl GRCh38 primary assembly (release 110, fetch_data.sh p6).

Delivered

Item	Module
FASTA access: plain/BGZF, `.fai`/`.gzi` read and built (identical to htslib's), index check, build detection, contig aliases, mmap genome cache	`formats/fasta.py`
BGZF writer, tabix `.tbi`, BCF 2.2 encoder, CSI index (pure Python)	`formats/bgzf.py`, `formats/bcf.py`
Plus-strand mapping with audit row per locus; indel resolution from `SourceSeq` (context score), left normalisation	`harmonize/sites.py`
Site grouping and the OUT-031 genotype-combination algorithm (ploidy-aware, Infinium I/II signal classes)	`harmonize/combine.py`
`simurg gtc-to-vcf` (per-sample VCF.gz + .tbi, `--merge` cohort BCF + .csi, audit Parquet)	`pipeline/vcf.py`
`simurg export bedgraph	finalreport

Acceptance (plan §18 P6)

bcftools validation and indexes

Every VCF passes bcftools view; bcftools index --nrecords on our .tbi equals the record count (635 796); region queries through our index (1:1-2M, X:150-156M, MT, Y, 22:30.0-30.1M) return exactly the records a full scan finds. The cohort BCF reads back identically to the per-sample VCFs (635 796 records × 3 samples, 0 differences in GT/GS/BAF/LRR) and its CSI index answers region queries.

Genotypes vs gtc2vcf (MIT), 36 samples, one record per probe (`--unsquash-duplicates`)

Both sides left-aligned with bcftools norm -f, compared per probe and sample as allele sets.

Category	Count (probe × sample)	Explanation
identical calls	23 656 010
identical no-calls	102 077
our merged multi-allelic site differs from the single-probe call	2 414	e.g. an Infinium II [G/T] probe calls GG while another probe at the site shows G/C: C shares G's dye, so the probe cannot see it — our site genotype is consistent with every probe
our merged site missing, probe called	3 945	the probes at the site contradict each other (`W_MERGE_INCONSISTENT`) or leave several genotypes (`W_MERGE_AMBIGUOUS`)
our merged site called, probe no-call	65	other probes at the site resolve it
het in a haploid context → missing (OUT-018)	14	male X non-PAR and MT heterozygous calls
female Y → missing (OUT-018)	11
indels gtc2vcf leaves unresolved (`REF=I/D`) that we resolve	144 probes	checked against GTCtoVCF below
unexplained differences	0

Locus sets: gtc2vcf writes 22 400 intensity-only loci that SimurgArray skips by design (OUT-019), and 49 indels it cannot resolve either (W_SKIP_INDEL_REF for us). Nothing we write is missing from gtc2vcf.

Genotypes vs GTCtoVCF (Apache-2.0), 36 samples, merged sites

(filled in below)

PLINK vs plink2 (GPL, subprocess oracle, built from source)

simurg export plink and plink2 --bcf <bcftools norm -m- of our BCF> --make-bed --split-par hg38 agree on all 637 397 variants × 3 samples (bed codes identical; the 1 601 extra variants are split multi-allelic sites).

BAF orientation, bedGraph

EDGE-017: at every called homozygous single-probe site where the probe's A allele is ALT, VCF BAF = 1 − GTC BAF (float32), otherwise unchanged (> 500 sites of each kind checked per sample).
EDGE-023: raw LRR bedGraph values equal the GTC values after %.6g formatting for every single-probe position.

Not yet implemented (listed in known_unknowns K-023)

--auxiliary-loci (OUT-035), --info-fields and --extra-format (OUT-012), --merge-snv-indel (OUT-021), PLINK 2 .pgen output.

3.8 KiB Raw Blame History Unescape Escape