SimurgArray/docs/research/00_phase0_report.md

SimurgArray — Phase 0 Research Report: can the reference array-analysis product be independently reimplemented?

Status: PHASE 0 (research only, no implementation code) · Research date: 2026-09-27 · Sanitized and brought into the repository afterwards (see README.md in this folder). Project: SimurgArray, a personal, non-commercial research toolkit. Canonical repository: https://gitea.yaylali.uk/kemal/SimurgArray. The roadmap and ground rules that came out of this study are in SIMURGARRAY_MASTER_PLAN.md (plan) and SIMURGARRAY_FUNCTIONAL_SPEC.md (spec).

Evidence base: R1_reference_product.md (product dossier), R2_open_source_ecosystem.md (GitHub ecosystem, source-level math with file:line refs), R3_science_formats_algorithms.md (formats + statistics), R4_datasets.md (public data), and the documentation walk-through files A_cli_inputs.md … E_release_history.md. Every claim below carries a tag; the URL / file:line sits in the referenced file section.

Tags: [DOC] public vendor documentation · [LIT] scientific literature/patents · [OSS] open-source implementation · [OBS] experimentally observed (none yet) · [INF] inferred · [HYP] hypothesis.

Naming: "the reference product" = the vendor's current Infinium array secondary-analysis software; refcli = its CLI binary; "ACLI" = the vendor's earlier array CLI (its predecessor); "the vendor cloud platform" = the vendor's hosted analysis and data platform.

---

Executive verdict

Question	Answer
Is a reference-compatible genotyping pipeline (IDAT → GTC → VCF + QC) feasible?	Yes, with high confidence. Formats are publicly specified [DOC/OSS]; the math of normalization, GenCall, BAF/LRR is reproduced in MIT-licensed open source that claims md5-identical GTCs vs ACLI [OSS: gtc2vcf]. The reference product itself does not cluster — it consumes Illumina EGT files [DOC].
Is numerical (bit-level) GTC compatibility achievable?	Probably, pending experiment. Whether the reference product's genotype call equals ACLI 2.1 output is unverified [HYP]. The practical target is the vendor-produced GTCs already published with 1000 Genomes (plan §16.2).
Can PGx CNV / star alleles / cyto CNV-LOH be cloned?	Not cloned — re-engineered as compatible alternatives. Algorithms and .dat model formats are undocumented [DOC gap]. Output schemas are documented and can be matched; results can only be matched semantically (same calls), not numerically.
Hardware acceleration?	The reference product is CPU-only, explicitly "no FPGA" [DOC]. Workload is I/O- and memory-bound linear algebra per sample; FPGA is not justified [INF]. GPU is optional only for large cohort re-clustering.
Main risk	At study time: legal/provenance, not science — (1) the vendor EULA family forbids deriving "underlying ideas, algorithms" from the Software [DOC]; (2) the best open reference (gtc2vcf) states it ports Illumina's GenCall C# code [OSS]. Resolved by the decisions below (plan §2): no vendor binary is run or inspected, and MIT code is reused openly with attribution.

---

1. What the reference product actually is

• The vendor's analysis suite for Infinium BeadChip data; CLI binary refcli, a self-contained .NET 9 app for linux-x64 / win10-x64; no ARM build [DOC; INF from RIDs]. Recommended 8 cores / 32 GB RAM [DOC].
• Lineage: AutoConvert → IAAP-CLI → ACLI (the vendor's earlier array CLI, retired 2024-12) → the reference product (v1.0 2023-12 … v1.4 2026-05) [DOC]. Stack traces in release notes show ArrayAnalysis.Core.* namespaces [DOC] ⇒ genotyping core is very likely inherited from ACLI [INF].
• Deployed locally (free, offline except star-allele) and on the vendor cloud platform [DOC].
• Not a clustering tool: custom EGT generation is still done in GenomeStudio [DOC].

2. Workflows

Workflow	Command	Local?	Algorithm disclosed?
Control-probe QC	qc call --array-type genotyping|methylation	yes	Formulas fully tabulated [DOC]
QC report	qc report (HTML + xlsx/csv, thresholds callRate 0.98, logRDev 0.20)	yes	yes (thresholding) [DOC]
Genotyping	genotype call (IDAT+BPM+EGT → GTC v5, cutoff 0.15)	yes	qualitatively [DOC]; quantitatively only via patents + OSS
VCF export	genotype gtc-to-vcf (→ *.snv.vcf.gz, GT:GS:BAF:LRR)	yes	yes — mirrors Illumina/GTCtoVCF (Apache-2.0) [DOC/OSS]
BedGraph	genotype gtc-to-bedgraph	yes	trivial
PGx CN	pgx copy-number train/call (≥24 samples batch, .dat model, 7 genes/10 regions)	yes	no [DOC gap]
PGx star alleles	pgx star-allele call/annotate (PharmVar/CPIC/DPWG)	yes, paid, license-server	no
Cytogenetics CNV/LOH/mosaic	cyto call/annotate (cyto_model.dat, ISCN output)	yes	no (outputs suggest HMM/segmentation) [HYP]
Methylation full QC (beta, M, detP)	cloud only	no	method named: Noob + nonlinear dye bias + negative-control detP [DOC]
Final Report / Locus Summary	cloud only	no	columns documented [DOC]

3. Inputs consumed [DOC]

IDAT pairs (Barcode_RxxCxx_Grn/Red.idat), BPM manifest, CSV manifest (SourceSeq, RefStrand), EGT cluster file, genome FASTA (+fai, contigs 1 not chr1), sample sheet (GenomeStudio-compatible), PGx CN model .dat + .msk, cyto model .dat, PGx database .zip, cyto annotation .zip, QC YAML.

4. Outputs produced [DOC]

Per sample: GTC v5; *.snv.vcf.gz+.tbi (VCFv4.1, FILTER always PASS, ploidy-adjusted GT, BAF/LRR oriented to REF=A/ALT=B → may be 1−GenomeStudio BAF); LRR/BAF bedgraphs; PGx *.cnv.vcf.gz (CN:NR, CN≤5, QUAL≤60); cyto *.cnv.vcf.gz (+ JSON ISCN annotation); star-allele JSON. Per batch: gt_sample_summary.csv/json (Autosomal Call Rate, Call Rate, LogRDev, Sex Estimate, TGA control), controls.raw_metrics.csv, controls.qc_metrics.csv, QC HTML/xlsx, cn_batch_summary.csv, five PGx CSVs.

5. What is publicly documented (sufficient to implement)

Area	Source
IDAT v3 binary layout (field codes, TOC)	illuminaio/BeadArrayFiles/gtc2vcf docs & code [OSS]
BPM, EGT, GTC v3–v5 layouts, incl. 52-byte normalization transform record	BeadArrayFiles + GTC_File_Format_v5.pdf [DOC/OSS]
Application of normalization: offset→rotate→shear→scale→clamp; θ=(2/π)atan2(y,x), R=x+y	BeadArrayFiles GenotypeCalls.py [DOC/OSS]
Normalization bins (norm_id + 100·assay_type) mod 256	BeadArrayFiles BeadPoolManifest.py:118-130 [OSS]
BAF / LRR from cluster centres (piecewise-linear), LRR=log2(R/R_exp)	Peiffer 2006 Genome Res [LIT]
TOP/BOT, PLUS/MINUS, indel [I/D] strand rules	Illumina strand tech note [DOC]
GenCall cutoff 0.15, GC10/GC50 definitions	tech notes [DOC]
Control-probe QC formulas (all ratios, offset 3000)	reference-product output docs [ref-docs:output-files] [DOC]
VCF conversion incl. multi-allelic merging, indel resolution	Illumina/GTCtoVCF + reference-product docs [DOC/OSS]
Normalization fit & GenTrain concepts	expired patents US7035740 (2024-05), US7467117 (2024-03); GenTrain 2/3 tech notes [LIT/DOC]
cnvPartition model	Illumina CNV tech note [DOC]

6. Implementable from published science (independent of vendor code)

Normalization fitting (patent + Teo 2007 description of the 400-homozygote approach) · Bayesian genotype calling in θ/R given EGT · BAF/LRR/LogRDev · sex inference (X het + Y intensity) · CNV/LOH/mosaic (PennCNV HMM [public domain], MoChA [MIT], cnvPartition tech note) · contamination (BAFregress, VerifyIDintensity model) · relatedness (KING-robust math) · methylation Noob/dye-bias/detP (Triche 2013, sesame) · star-allele matching (PharmCAT/PyPGx logic) · own cluster training (GMM / t-mixture à la Illuminus/CRLMM).

7. Behaviours that cannot be read from documentation

At study time these were listed as "requires black-box characterization of the vendor binary". Per decision D2 below, the vendor binary is not run; each item is instead resolved from published vendor-produced GTCs, open oracles and synthetic experiments, or recorded as a known unknown in docs/known_unknowns.md.

1. Is the reference product's genotype call GTC ≡ ACLI GTC (md5 with timestamps masked)? — not testable without the binary; bit-exact target is the published vendor-produced GTCs instead.
2. Exact GenCall score mapping & edge cases (low bead counts, zero intensities, Omni5 normID overflow).
3. Sex Estimate, Autosomal Call Rate locus set, LogRDev locus set/trimming.
4. VCF details: REF/ALT choice for indels, ploidy rules on X/Y/MT, GS Phred scaling, BAF orientation flip.
5. PGx CN: CN calls vs regions, NR distribution, QUAL calibration (targets for our own model).
6. Cyto CNV/LOH: event boundaries, MF estimates, filters (targets for our own caller).
7. Star alleles: diplotype/score behaviour on known samples.

8. Hardware-specific parts

None. CPU-only by vendor statement [DOC]. The product brand is FPGA-associated for NGS, but the array product is not [DOC]. ⇒ No FPGA work in scope; revisit only if profiling shows a hotspot (unlikely: per-sample work is ~O(#probes) ≈ 1–5 M floats).

9. Open-source projects (see R2 table for ~45 entries)

Project	License	Role in SimurgArray
Illumina/BeadArrayFiles	BSD-2	format spec + oracle (GTC/BPM/EGT readers); may be ported with attribution
Illumina/GTCtoVCF	Apache-2.0 (archived)	VCF conversion reference + oracle; may be ported with attribution
freeseek/gtc2vcf (idat2gtc, gtc2vcf, BAFregress)	MIT	test oracle and algorithm reference; may be ported with attribution
broadinstitute/picard picard.arrays	MIT	second independent format reader; QC metric definitions (WARP)
freeseek/mocha	MIT	mosaic CNV reference
WGLab/PennCNV	Unlicense	CNV HMM; PFB/GC model
PharmGKB/PharmCAT	MPL-2.0	star-allele oracle (subprocess)
sbslee/pypgx	MIT	array-VCF star allele reference
bcftools	MIT	VCF I/O oracle, +fixref
plink2, verifyIDintensity, GenotypeHarmonizer, illuminaio	GPL	subprocess oracles only, never linked or copied
zCall, argyle, Aldy, Cyrius	none / NC / PolyForm	excluded (ideas from their papers only)

Gap: no open-source GenTrain / EGT writer exists [OSS survey].

10. Realistically cloneable (target: numerical compatibility)

IDAT/BPM/EGT/CSV readers · GTC v5 writer · normalization · GenCall calling+score · BAF/LRR/LogRDev · sex estimate · call rates · control-probe QC · GTC→VCF · bedgraph · gt_sample_summary · QC report table.

11. Implement as compatible alternatives (target: semantic compatibility, same output schema)

Cluster-file generation (own "ArrayTrain", explicitly not GenTrain) · PGx CNV (own GC-corrected batch model) · star alleles (PharmVar-based solver, PharmCAT/PyPGx as oracles) · cyto CNV/LOH/mosaic (HMM on LRR+BAF) · methylation QC (Noob etc.) · contamination/relatedness (not in the reference product; value-add).

12. Proposed architecture (as sketched in Phase 0; the binding layout is plan §4)

simurg/
  core/            canonical data model (Arrow schemas: Probe, Locus, SampleIntensity, Call, Provenance)
  formats/         idat/ bpm/ csv_manifest/ egt/ gtc/ samplesheet/ vcf/ plink/ bedgraph/
  signal/          raw_xy (probe→locus assembly, bead-count rules), normalization/ (fit, apply), polar
  genotype/        gencall/ (EGT-based caller), train/ (ArrayTrain: own clustering), rescue/ (optional)
  cn/              baf_lrr/, gc_wave/, cyto_hmm/, pgx_cn/
  pgx/             star_allele/ (PharmVar db builder, solver), annotate/
  qc/              controls/, sample/ (call rate, LogRDev, sex), contamination/, relatedness/, report/
  harmonize/       strand (TOP/BOT/PLUS), REF/ALT, build liftover hooks, audit log of every flip
  pipeline/        DAG runner, resumable, per-step provenance (hashes, versions, params, seed)
  cli/             `simurg qc|genotype|gtc-to-vcf|cn|cyto|pgx|pipeline` (reference-like verbs)
  compat/          reference-CLI option aliases + output-name compatibility layer
  difftest/        oracle adapters (published vendor GTCs, gtc2vcf, BeadArrayFiles, GTCtoVCF, PennCNV, PharmCAT)
  synth/           synthetic IDAT/intensity generator (ground-truth genotypes, CN, contamination, dye bias)
  bench/  tests/  docs/

Principles: science modules are pure functions over Arrow/NumPy arrays; no I/O inside algorithms; every transform returns data plus an audit record. Reference implementation in Python (NumPy/SciPy, pyarrow, pysam/cyvcf2); performance tier later in Rust (PyO3) for IDAT parsing + normalization + calling — the hot, embarrassingly-parallel per-sample path.

13. Phased plan (as proposed in Phase 0; the binding roadmap and gates are plan §18)

Phase	Deliverable	Exit gate (measurable)
0 Research (this)	research docs, decisions D1–D3, dataset fetch scripts	decisions taken (below); datasets checksummed
1 Parsers	IDAT, BPM, EGT, CSV, GTC read/write, sample sheet	100% field equality vs BeadArrayFiles + illuminaio on ≥254-series GEO sweep + Omni2141; GTC round-trip byte-identical
2 Canonical model	Arrow schemas, locus assembly (Inf I/II, bead count ≥2 rule)	raw X/Y identical to GTC RAW_X/RAW_Y (TOC 1000/1001) for 1000G Broad GTCs
3 Normalization	fit + apply per bin	13-float transforms vs Illumina GTC TOC 400:
4 Genotyping (EGT-based)	GenCall-compatible caller + score, BAF/LRR, sex	vs Illumina GTCs: genotype concordance ≥99.99%, GC score
4b ArrayTrain	own clustering → EGT	vs Illumina EGT calls on Omni2141: ≥99.5% concordance; vs 30x WGS non-ref discordance ≤ GenCall's
5 QC	controls, sample summary, QC report	control metrics equal the documented formulas; Sex Estimate 100% on 1000G known sex
6 Outputs	VCF (reference-compatible layout), PLINK, bedgraph, JSON/CSV summaries	bcftools field-level diff vs gtc2vcf/GTCtoVCF: 0 unexplained discrepancies
7 Differential testing	automated EXACT/NEAR/DISCREPANCY/UNKNOWN dashboard	nightly on demo + Omni subsets
8 Validation	science validation vs WGS truth, trios, replicates	concordance, Mendelian error, reproducibility reported
8b CN/Cyto/PGx	own CN HMM, PGx CN, star alleles	CNV vs the vendor's public 1000 Genomes WGS reanalysis (AWS Open Data) & PennCNV; CYP2D6 CN vs WGS truth; star alleles vs PharmCAT-on-WGS
9 Optimization	Rust core, mmap, multithreading	throughput at least on par with the vendor's documented figure (35 s/sample VCF claim) on a comparable x86 box
10 Packaging	conda/pip/Docker, docs, CI	reproducible builds, provenance manifest per run

14. Validation strategy

Three independent axes, never conflated:

1. Format compatibility — parses/writes files other tools accept (BeadArrayFiles, bcftools, plink2).
2. Numerical compatibility — field-level diff vs oracles with explicit tolerances (float32 ULP-aware). Oracles: (a) Illumina GTCs already published in the 1000G Broad set (the vendor ground truth, no licensed binary needed), (b) GenomeStudio FinalReport in the 1000G Sanger set, (c) gtc2vcf (MIT), (d) BeadArrayFiles / GTCtoVCF / Picard. Outputs of the reference product's binary are not an oracle (decision D2).
3. Scientific validity — vs truth independent of the vendor: NYGC 30x / GIAB v4.2.1 (concordance, NRD, sensitivity/precision per genotype class & MAF bin), 337 Omni trios + GDA-demo trios (Mendelian error), GSE165845 triplicates (reproducibility), the vendor's public 1000 Genomes WGS reanalysis (AWS Open Data) + GIAB SV (CNV), synthetic data (exact truth). Reporting: EXACT / NEAR (within tolerance) / DISCREPANCY / UNKNOWN per field, per locus class (Inf I/II, indel, X/Y/MT, PAR, low-intensity).

15. Datasets required (all verified public, see R4)

Priority	Dataset	Use
P0	GDA v1 Demo 36 (465 MB; 3 CEU trios + replicates, incl. NA12878)	smallest end-to-end fixture
P0	GSA v4 ePGx Demo 36 (872 MB) + BPM/CSV/EGT + CN/cyto models	primary reference-compatibility target (inputs and documented schemas)
P0	Bioconductor IlluminaDataTestFiles, hapmap370k (Artistic-2.0)	only fixtures we may vendor in CI
P1	1000G Broad Omni2.5 IDAT+GTC 2,141 samples (107 GB) + BPM/EGT/CSV	GenCall oracle w/o licensed software; trios; clustering training
P1	1000G Sanger Omni2.5-8 (6.2 GB) + GenomeStudio FinalReport	GenomeStudio-output oracle
P1	GEO GSE165845 GSA 120×3 (45 GB)	reproducibility, GSA-class concordance
P1	NYGC 30x 1kGP VCFs, GIAB v4.2.1, the vendor's public 1000 Genomes WGS reanalysis (AWS Open Data) (CNV/PGx)	truth
P2	GSE96909 (NA12878 on 9 designs), GDA ePGx 480, OneK1K GSA 1,104, 254 GEO series	cross-design, scale, parser fuzzing

Vendor product files/demo data: fetch-with-checksum script only, never committed or redistributed (site terms; plan §2 rule 4).

16. Experiments (each: input / variable / observable / confidence)

Experiments that needed the vendor binary (original E1 "GTC md5 of refcli vs ACLI vs gtc2vcf", and the refcli side of E4–E8, E11) are dropped under decision D2. The rest run against published vendor-produced GTCs, open oracles and synthetic data:

E1 (dropped: required the vendor binary) — replaced by E2 against published GTCs. E2 Normalization transform per bin: SimurgArray vs Illumina GTC (1000G Broad) — tolerance study. E3 Synthetic single-SNP perturbation in IDAT (shift X/Y, bead count 0/1/2, NaN, extreme) → call/score response of SimurgArray vs gtc2vcf. E4 Subset-of-samples invariance: is genotype calling strictly per-sample? (expect yes [INF]; check on published GTCs.) E5 Sex Estimate decision boundary: sweep X-het and Y-R in synthetic IDATs (our definition; documented thresholds). E6 LogRDev / Autosomal call-rate locus sets: remove chromosomes/loci classes, compare with published GTC metrics. E7 VCF: indel REF/ALT, duplicates, MAV merging, --use-infI-nc-info, ploidy on X/Y/MT/PAR — vs GTCtoVCF/gtc2vcf and the documented reference behaviour. E8 PGx CN: batch-size and batch-composition sensitivity (24→500), GC-correction on/off — our own model. E9 Cyto: spiked-in synthetic LRR/BAF events (size, mosaic fraction 5–100%, LOH) → detection curves. E10 Star alleles: known 1000G samples with WGS-derived truth (PharmCAT-on-WGS as reference). E11 Determinism: repeat runs, thread counts, OS/architecture (x86-64 vs aarch64) for SimurgArray itself (the vendor reports tiny cross-platform PGx differences [DOC]).

17. Known unknowns (tracked in docs/known_unknowns.md)

U1 reference-product GTC ≡ ACLI GTC? (not testable under D2) · U2 GenCall score mapping details beyond OSS · U3 Sex Estimate algorithm · U4 LogRDev/Autosomal CR locus set · U5 PGx CN algorithm & .dat format · U6 cyto caller & .dat format · U7 star-allele scoring/solver · U8 GenTrain internals (NN cluster imputation, scoring weights, 14–15 unused EGT floats) · U9 GTC PDF says Raw X = green vs OSS red — test · U10 IDAT MidBlock int32 vs uint8 · U11 EGT intensity-threshold version gate (≥7 vs 9) · U12 reference-product EULA text (not public; moot under D2).

18. Risk register (updated to the decisions taken)

#	Risk	L	I	Mitigation / status
R1	EULA prohibits deriving algorithms from the vendor binary	—	—	Closed by D2: the vendor binary is never run, benchmarked, inspected or decompiled; oracles are published vendor-produced GTCs
R2	Provenance: gtc2vcf ports Illumina GenCall C# code	L	M	D1: MIT code may be ported with attribution in THIRD_PARTY_NOTICES.md; no strict clean room is required for a personal research project
R3	Research notes (R2) contain gtc2vcf-derived details	L	L	Closed by D1: allowed; cite sources (plan §7 tags)
R4	GenTrain can't be matched ⇒ dependence on Illumina EGT files (non-redistributable)	H	M	ArrayTrain own clusters; user supplies EGT; fetch script
R5	Illumina manifest/EGT/demo licensing (non-commercial, no redistribution)	H	M	never commit; checksum fetcher; CI on Artistic-2.0 fixtures + synthetic data
R6	PGx/cyto calls diverge from the reference product	H	M	"compatible schema, independently validated"; publish concordance vs truth, never claim equivalence
R7	Encrypted v1 IDATs (DES, only GPL illuminaio reads them)	L	L	out of scope; detect and error clearly
R8	Floating-point non-determinism across platforms/threads	M	M	deterministic reductions, fixed ordering, float32/64 policy per field, ULP tolerances
R9	Trademark misuse	M	M	project named SimurgArray; docs refer to Illumina file formats only and use no vendor product brand (plan §2 rule 5)
R10	Scale (100k samples) memory	M	M	per-sample streaming; only clustering needs cohort-wide data (column-chunked Arrow)
R11	Vendor binary needs x86-64; dev box is aarch64	—	—	Moot under D2 (the binary is not used)

Decisions taken (see plan §2)

• D1 — Provenance policy: taken. SimurgArray is a personal, non-commercial research project. MIT / BSD / Apache-2.0 / Unlicense code (gtc2vcf/idat2gtc/BAFregress, mocha, BeadArrayFiles, GTCtoVCF, Picard picard.arrays, PennCNV, PyPGx, bcftools) may be ported or vendored with attribution in THIRD_PARTY_NOTICES.md. GPL tools (plink2, illuminaio, verifyIDintensity, GenotypeHarmonizer, akt) are external subprocess oracles only, never copied or linked. Code with no licence or non-commercial/PolyForm terms (zCall, argyle, Aldy, Cyrius, Stargazer, QuantiSNP binaries) is not used; ideas from their papers are fine. The Phase-0 "Path B strict clean room" option is not needed.
• D2 — Black-box access: taken. Black-box runs of the vendor binary are not part of the plan, and vendor software is never decompiled, disassembled or inspected, nor licence checks bypassed. Oracles are the published vendor-produced GTCs (1000 Genomes Omni2.5 Broad set; GenomeStudio FinalReport in the Sanger set), gtc2vcf and BeadArrayFiles (plus GTCtoVCF/Picard for formats). Reference-product behaviour is taken only from its public documentation (files A–E); what cannot be settled that way is logged as a known unknown.
• D3 — Scope of v1.0: taken. Genotyping + QC + VCF/PLINK first (plan phases P0–P8, then v1.0); ArrayTrain, CN/LOH/mosaic, PGx and methylation control QC are v2 modules (plan §18).