kemalandClaude Opus 5.5
0861b27c2b
Research notes R1-R4, documentation-study files A-E (source of the functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use neutral naming and current project decisions (plan §2). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
447 lines
37 KiB
Markdown
447 lines
37 KiB
Markdown
# R2 — Open-source ecosystem for Illumina Infinium genotyping-array analysis
|
||
|
||
> **Archived Phase-0 study (sanitized copy, see `README.md`).** Written 2026-09-27, before the decisions in
|
||
> `SIMURGARRAY_MASTER_PLAN.md` §2. The analysed vendor product is called "the reference product" and its CLI
|
||
> `refcli`; its documentation URLs are replaced by `[ref-docs:…]` tokens. Where this file speaks of a clean room,
|
||
> legal clearance, or running/benchmarking the vendor binary, plan §2 supersedes it: personal research; MIT/BSD/Apache
|
||
> code may be reused with attribution; GPL tools are subprocess oracles only; vendor binaries are never run for
|
||
> comparison, inspected or decompiled; oracles are published vendor-produced GTCs, gtc2vcf and BeadArrayFiles.
|
||
|
||
Survey date: 2026-09-27. Scope: projects relevant to building SimurgArray, an independent reimplementation of the reference product
|
||
(IDAT → normalization → GenCall → GTC → VCF, BAF/LRR, CNV, sex, contamination, PGx star alleles).
|
||
|
||
Local shallow clones used for this report live in `<repo>/refs/`:
|
||
`BeadArrayFiles`, `GTCtoVCF`, `gtc2vcf`, `mocha`, `illuminaio`, `picard` (sparse: `src/main/java/picard/arrays`),
|
||
`PennCNV`, `zCall`, `verifyIDintensity`, `AGCT`. All `file:line` references below point at those clones.
|
||
Commit dates are from `git log -1` on the clones, or from the GitHub API (`pushed_at`) where noted.
|
||
|
||
---
|
||
|
||
## 0. Key findings (TL;DR)
|
||
|
||
1. **freeseek/gtc2vcf `idat2gtc.c` (MIT, C, active: last commit 2026-02-26) is a complete open reimplementation of
|
||
Illumina AutoCall/GenCall for an existing cluster file.** It covers IDAT parsing, per-bin affine normalization
|
||
(versions 1.1.2 = GenTrain 2.0 and 1.2.0 = GenTrain 3.0), the GenCall Bayesian/fuzzy call and score, gender
|
||
estimation, BAF/LRR, LogR dev, GC10/GC50 and writing GTC v3/v5. The author says the output is "equivalent to the
|
||
output of the Illumina GenCall algorithm". With `--autocall-date ""` and a `--preset` (1 = AutoConvert 1.6.3.1,
|
||
2 = AutoConvert 2.0.1.179, 3 = IAAP CLI, 4 = ACLI) he says you "should be able to verify that you get
|
||
the same md5sum" as patched, deterministic Illumina binaries (README.md:233-244). This is **byte-identical GTC**,
|
||
a stronger claim than genotype concordance. It does **not** implement GenTrain *clustering*, so it cannot create
|
||
an EGT.
|
||
- **Provenance note for clean-room purposes:** the header (idat2gtc.c:27-30, 63-67) says it "reimplements
|
||
functionalities and ideas present in AutoConvert (v1.6.3.1), GTCtoVCF, BeadArrayFiles" and cites Illumina's
|
||
GenCall C# source code (`NormalizationInfinium.cs`, `GenTrain60.cs`, `Utils.cs`). It says these were "provided
|
||
by Illumina without license restrictions". It also deliberately **re-implements Illumina bugs** for bit-exactness
|
||
(idat2gtc.c:95-106). It is MIT-licensed, so it is legally reusable with attribution. However, it is effectively a
|
||
port of Illumina code. If SimurgArray needs a strict clean room, route it through a spec-writer/implementer split (see §6). *(Not required: plan §2 allows MIT reuse with attribution.)*
|
||
- The key patents have expired: US7035740 (normalization) on 2024-05-05 and US7467117 (clustering/genotyping) on
|
||
2024-03-24 (idat2gtc.c:47-51).
|
||
2. **Illumina/BeadArrayFiles is NOT Apache-2.0.** The README carries a **BSD-2-Clause ("FreeBSD") license**,
|
||
Copyright (c) 2020 Illumina (README.md:84-111). The repo has no LICENSE file, so the GitHub API reports no license.
|
||
Last commit 2025-01-30. It is the reference parser for GTC/BPM/EGT and holds the canonical normalization
|
||
*application* (`NormalizationTransform.normalize_intensities`). It does not hold the *estimation* of the transform.
|
||
3. **Illumina/GTCtoVCF is Apache-2.0 and ARCHIVED** (GitHub `archived: true`). Its README says "Semi-archived … As of May
|
||
2023, it is now recommended to use … [product]". It only supports Python 2.7. It is still useful as an algorithm
|
||
reference for indel/reference-allele resolution from manifest source sequences, and for the VCF FORMAT layout
|
||
(GT, GS as Phred, BAF, LRR).
|
||
4. **broadinstitute/picard `picard.arrays` (MIT, Java, active)** is a second, independent, permissively licensed set of
|
||
GTC/BPM/EGT/manifest readers. It also includes `GtcToVcf`, a normalized-intensity computation, the conversion of
|
||
cluster (Theta, R) mean/dev to X/Y, and QC metric tools (BafRegress, VerifyIDIntensity, CollectArraysVariantCallingMetrics).
|
||
It is used in Broad **WARP**'s production Illumina genotyping pipeline (BSD-3-Clause).
|
||
5. **No open-source GenTrain clustering (EGT writer) exists.** Searches found only readers. SimurgArray will have to build
|
||
cluster training from the Illumina GenTrain 2/3 technical notes and the patents.
|
||
6. **Licensing hazards:** illuminaio (GPL-2), plink2 (GPL-3), GenotypeHarmonizer/systemsgenetics (GPL-3),
|
||
verifyIDintensity (GPL-3), argyle (CC BY-NC-SA 4.0), Aldy (non-commercial IU license), Cyrius (PolyForm Strict,
|
||
no commercial use), zCall (no license = all rights reserved). Use these only as external executables or
|
||
test oracles, never as linked code.
|
||
|
||
---
|
||
|
||
## 1. Summary table
|
||
|
||
Reuse legend. **DEP** = safe as a linked dependency. **ALG** = algorithm reference. **ORACLE** = run as a test oracle
|
||
(black box). **IGN** = ignore.
|
||
|
||
| Project | URL | License | Lang | Last activity | What it implements | Faithfulness to Illumina | SimurgArray reuse | License risk |
|
||
|---|---|---|---|---|---|---|---|---|
|
||
| gtc2vcf / idat2gtc / BAFregress | github.com/freeseek/gtc2vcf | **MIT** (LICENSE; per-file MIT headers) | C (bcftools plugins) | 2026-02-26 (v2026-01-26) | IDAT/BPM/EGT/GTC/CSV parsing; normalization 1.1.2 and 1.2.0; GenCall calling and score; gender; BAF/LRR; GTC writer; GTC→VCF; cluster re-centering; BAFregress contamination | Claims md5-identical GTC vs AutoConvert, AutoConvert 2.0, IAAP-CLI and ACLI when timestamps are disabled | **ALG + ORACLE**; DEP is possible (MIT) | Low legally. Clean-room provenance caveat (port of Illumina C#) |
|
||
| mocha | github.com/freeseek/mocha | **MIT** | C | 2025-08-22 | Mosaic chromosomal alteration HMM on BAF/LRR/phase; BAF cluster median adjustment; LRR GC-wave regression | N/A (downstream) | ALG (CNV/mCA module) | Low |
|
||
| BeadArrayFiles | github.com/Illumina/BeadArrayFiles | **BSD-2-Clause** (in README; no LICENSE file) | Python 3 | 2025-01-30 | GTC v3-5, BPM v3-5 (locus v4,7,8), EGT v3 (data-block v<5, 6-8, 9) readers; `NormalizationTransform`; LocusAggregate; report examples | Official Illumina | **DEP / ALG / ORACLE** (format spec: docs/GTC_File_Format_v5.pdf) | Low (keep notice) |
|
||
| GTCtoVCF | github.com/Illumina/GTCtoVCF | **Apache-2.0** | Python 2.7 | Archived (last code 2024-03-15) | GTC+BPM+CSV+FASTA → VCF; indel ref-allele resolution; GS as Phred; BAF/LRR FORMAT | Official, superseded by the reference product | ALG (VCF conversion, indel logic); ORACLE | Low (NOTICE requirement) |
|
||
| Picard `picard.arrays` | github.com/broadinstitute/picard | **MIT** | Java | 2026-07-31 | InfiniumGTCFile, IlluminaBPMFile, InfiniumEGTFile, normalization manifest, GtcToVcf, extended manifest, BafRegress and VerifyIDIntensity metric wrappers, arrays calling metrics, VcfToAdpc | Production use in WARP; reads Illumina-produced GTC (does not call) | ALG + ORACLE; DEP if JVM is acceptable | Low |
|
||
| WARP Illumina Genotyping Array | github.com/broadinstitute/warp | BSD-3-Clause | WDL | active 2026-09 | Production pipeline: IAAP-CLI → Picard GtcToVcf → metrics, VerifyIDIntensity, BafRegress, zCall, fingerprinting | Uses Illumina binaries | ALG (pipeline/QC design), output spec | Low |
|
||
| bcftools | github.com/samtools/bcftools | MIT/Expat (GPL if built with GSL) | C | 2026-09-21 | Host for gtc2vcf plugins; `+fixref`, `norm`, `+split-vep` | — | DEP (subprocess) or ORACLE | Low if built without GSL |
|
||
| illuminaio | github.com/HenrikBengtsson/illuminaio (Bioconductor) | **GPL-2** | R + C (DES) | 2024-05-12 | IDAT reader (non-encrypted binary and **encrypted DES/XML IDATs**), BPM, BGX | Well tested | ALG only (clean re-derivation of encrypted IDAT) | **High** if linked |
|
||
| illumina_array_reader | github.com/a-xavier/illumina_array_reader | MIT | Rust | 2023 | IDAT reader (illuminaio port) → polars | — | DEP candidate (Rust) | Low. It is a port of GPL code, so verify provenance |
|
||
| IlluminaIdatFiles.jl | github.com/oheil/IlluminaIdatFiles.jl | MIT (per repo) | Julia | — | IDAT reader | — | IGN / ALG | Low |
|
||
| methylprep / methylcheck | github.com/FoxoTech/methylprep | MIT | Python | 2023-08 (stale) | IDAT parsing (methylation), NOOB, QC | Methylation only | ALG (IDAT parsing) | Low |
|
||
| mepylome | github.com/brj0/mepylome | MIT | Python/C++ | 2025 | Fast IDAT parsing (methylation), CNV from methylation arrays | — | ALG | Low |
|
||
| minfi | bioconductor/hansenlab/minfi | Artistic-2.0 | R | 2024-04 | Methylation arrays | — | IGN | Medium |
|
||
| sesame | github.com/zwdzwd/sesame | MIT + file | R | 2026-09-23 | Methylation arrays, pOOBAH, IDAT via own reader | — | IGN (methylation) | Low |
|
||
| crlmm | Bioconductor | Artistic-2.0 | R/C | Bioc release | CRLMM genotype calling for Illumina/Affy; CNV | Alternative caller; published concordance vs GenCall (Ritchie 2011) | ALG (alt. caller) | Medium |
|
||
| argyle | github.com/andrewparkermorgan/argyle | **CC BY-NC-SA 4.0** | R | 2018-03 (dormant) | Mouse MUGA arrays; reads GenomeStudio FinalReport; QC; BAF/LRR | — | IGN / ALG | **High** (non-commercial) |
|
||
| zCall | github.com/jigold/zCall | **none** (all rights reserved) | Python | 2015-02 | Rare-variant re-caller using z-score thresholds on normalized X/Y after GenCall | Post-GenCall rescue | ALG (idea only) | **High** (no license) |
|
||
| optiCall | opticall.bitbucket.io (Sanger) | Sanger terms (not OSI; binary + source on request) | C++ | ~2012 | Rare/common-variant caller using Illumina-normalized intensities | Alt. caller | ALG (paper only) | High |
|
||
| Illuminus | github.com/wtsi-npg/Illuminus | unclear/none in repo | C | stale | Mixture-model caller (Teo 2007). Paper documents Illumina normalization (400 homozygotes) | Alt. caller | ALG (paper) | High (unclear) |
|
||
| GenoSNP | Oxford Univ Innovation licence | Academic licence | C | stale | Within-sample VB caller | Alt. caller | IGN | High |
|
||
| PennCNV | github.com/WGLab/PennCNV | **Unlicense (public domain)** (UNLICENSE file) | Perl + C (kext HMM) | 2024-05-19 | CNV HMM on LRR/BAF with PFB and GC model; `split_illumina_report.pl`, `compile_pfb.pl`, `cal_gc_snp.pl` | De-facto standard for array CNV | **DEP / ALG** (HMM, PFB, GC-wave model) | **None** |
|
||
| QuantiSNP | (Oxford, binary) | Academic binary licence | MATLAB runtime | stale | OHMM CNV | — | IGN | High |
|
||
| iPattern | (Broad/Harvard) | Academic licence | R/Perl | stale | Intensity-pattern CNV | — | IGN | High |
|
||
| EnsembleCNV | github.com/HaoKeLab/ensembleCNV | none visible | R/Perl | ~2019-2021 | Meta-caller over PennCNV/QuantiSNP/iPattern; CNVR genotyping | — | ALG (ensemble idea) | High (no license) |
|
||
| GADA | Bioconductor/CRAN "gada" | GPL | R/C | stale | Sparse Bayesian segmentation of LRR | — | ALG | High if linked |
|
||
| ASCAT | github.com/VanLoo-lab/ascat | GPL-3 (per repo) | R | 2026-02 | Allele-specific CN for tumours from BAF/LRR | — | ALG | High if linked |
|
||
| plink2 | github.com/chrchang/plink-ng | **GPL-3** (2.0/COPYING) | C++ | 2026-09-27 | Genotype QC, sex check, relatedness (KING-robust), PCA | — | ORACLE (subprocess) | **High** if linked |
|
||
| bcftools +fixref | samtools/bcftools | MIT | C | active | Strand/REF fixing vs reference | — | DEP / ORACLE | Low |
|
||
| snpflip | github.com/biocore-ntnu/snpflip | MIT per PyPI (no LICENSE in repo) | Python | Archived 2019 | Detect reverse/ambiguous-strand SNPs | — | IGN / ALG | Low-Med |
|
||
| Will Rayner strand files | chg.ox.ac.uk/~wrayner/strand/ | Data, no explicit licence | data + Perl | maintained | Per-chip strand/build files (TOP/source strand → + strand) | Community standard | ORACLE (strand truth) | Medium (data terms unclear) |
|
||
| GenotypeHarmonizer | molgenis/systemsgenetics | **GPL-3** | Java | 2026-03 | Strand alignment via LD with reference | — | ORACLE | High if linked |
|
||
| conform-gt | faculty.washington.edu/browning | Apache-2.0 (reported) | Java | stale | Conform VCF to reference panel (strand/allele) | — | ORACLE | Low |
|
||
| AGCT | github.com/molgenis/AGCT | (none visible) | shell | — | IDAT→GTC pipeline using IAAP | — | IGN | — |
|
||
| peddy | github.com/brentp/peddy | MIT | Python | 2026-02 | Pedigree/sex/ancestry QC from VCF | — | DEP/ALG | Low |
|
||
| somalier | github.com/brentp/somalier | MIT | Nim | 2026-09-25 | Relatedness/sample swap from sites, works on array VCF | — | DEP/ALG | Low |
|
||
| KING | kingrelatedness.com | Binary, free download; source by request | C++ | — | Kinship | — | ORACLE | Medium |
|
||
| akt | github.com/Illumina/akt | GPL-3 (per repo) | C++ | stale | Ancestry and kinship tools | — | ORACLE | High if linked |
|
||
| verifyIDintensity | github.com/gjun/verifyIDintensity | **GPL-3** | C++ | 2015-05 | Contamination from array intensities (Jun 2012) | Paper method | ALG (re-derive); ORACLE | High if linked |
|
||
| BAFregress | inside gtc2vcf (`BAFregress.c`) | MIT | C | 2025-08 | Contamination via BAF regression on hom-calls (Jun 2012) | Same model as original | **DEP/ALG** | Low |
|
||
| PharmCAT | github.com/PharmGKB/PharmCAT | **MPL-2.0** | Java | 2026-09-25 | Star-allele matcher from VCF (array or WGS), CPIC phenotypes | Benchmarked on GSA/PGx arrays | ORACLE / DEP (file-level copyleft only) | Medium (MPL is file-level) |
|
||
| PyPGx (Stargazer successor) | github.com/sbslee/pypgx | MIT | Python | 2026-08 | Star alleles from NGS **and chip VCFs** (supports array input) | Benchmarked on GSA | ALG / ORACLE | Low |
|
||
| Stargazer | stargazer.gs.washington.edu | Academic, non-commercial | Python | legacy | Star alleles (NGS) | — | IGN | High |
|
||
| Aldy | github.com/0xTCG/aldy | **IU non-commercial licence** | Python/C | 2026-04 | Star alleles (NGS; VCF input) | — | IGN / ORACLE only | **High** |
|
||
| Cyrius | github.com/Illumina/Cyrius | **PolyForm Strict 1.0.0** | Python | 2023-10 | CYP2D6 from WGS | — | IGN | **High** |
|
||
| BeadArrayPython3 | github.com/dbrami/BeadArrayPython3 | (fork of BSD-2) | Python | — | Py3 port of BeadArrayFiles | — | IGN (upstream is Py3 now) | Low |
|
||
|
||
"Parses [product] outputs / 'refcli'": no public GitHub repo was found that parses reference-product-specific outputs
|
||
(the CNV VCF, PGx JSON/TSV, or `refcli` CLI logs). The reference product's GTC is v5, and Illumina's docs point to
|
||
BeadArrayFiles for parsing it. The SNV VCF FORMAT fields (GT, GS, BAF, LRR) are compatible with what
|
||
gtc2vcf/GTCtoVCF/Picard emit. So these three are the closest analogues for output-schema compatibility tests.
|
||
|
||
---
|
||
|
||
## 2. BeadArrayFiles (Illumina) — formats and normalization application
|
||
|
||
**License (corrected):** BSD-2-Clause text in `BeadArrayFiles/README.md:84-111` ("Copyright (c) 2020, Illumina … FreeBSD
|
||
Project"). It is **not** Apache-2.0. Apache-2.0 applies to GTCtoVCF (`GTCtoVCF/LICENSE`).
|
||
|
||
### 2.1 NormalizationTransform (GenotypeCalls.py:625-698)
|
||
- Binary layout, 52 bytes per transform (`read_normalization_transform`, :641-652): `int32 version`, then
|
||
`float32 offset_x, offset_y, scale_x, scale_y, shear, theta` (:637-639). The remaining 6 floats are reserved.
|
||
In gtc2vcf they are `cvx, cvy, nn12, rr12, taa, tbb` (idat2gtc.c:902-916).
|
||
- `normalize_intensities(x, y, threshold=True)` (GenotypeCalls.py:668-698):
|
||
```
|
||
if x <= 0 and y <= 0: return (nan, nan)
|
||
tx = x - offset_x ; ty = y - offset_y # translate
|
||
tx2 = cos(theta)*tx + sin(theta)*ty # rotate by -theta
|
||
ty2 = -sin(theta)*tx + cos(theta)*ty
|
||
tx3 = tx2 - shear*ty2 # shear (x only)
|
||
xn = tx3/scale_x ; yn = ty2/scale_y # scale
|
||
if threshold: xn = max(xn,0); yn = max(yn,0) # clamp at 0
|
||
```
|
||
- `rect_to_polar(x, y)` (:654-666): `R = x + y`, `theta = atan2(y, x) * 2/pi`. It returns `(nan, nan)` if `x == y == 0`.
|
||
- `get_normalized_intensities(normalization_lookups)` (:570-582) applies transform `lookups[i]` per locus.
|
||
|
||
### 2.2 Normalization-bin lookup (BeadPoolManifest.py:90-130)
|
||
- A per-locus `normalization_id` byte is read (it must be < 100, :91-94). Then
|
||
`norm_id += 100 * assay_type` and `norm_id %= 256` (:118-124). The mod is commented "To mimic the byte-wrapping
|
||
behavior from GenomeStudio, AutoCall, IAAP". This affects Omni5 (see GTCtoVCF changelog 1.2.1).
|
||
- Transform index = rank of `norm_id` among the sorted set of distinct ids (:125-130). The GTC stores transforms in
|
||
that order.
|
||
- `assay_type`: 0 = Infinium II, 1 = Infinium I A/T (red), 2 = Infinium I G/C (green). gtc2vcf infers it from probe and
|
||
source sequences (idat2gtc.c:1242-1272).
|
||
|
||
### 2.3 EGT reader (ClusterFile.py:83-240)
|
||
- Header: `int version (must be 3)`, strings `gencall_version, cluster_version, call_version, normalization_version,
|
||
date_created`, a byte `is_wgt (must be 1)`, string `manifest_name`, `int data_block_version`, string `opa`,
|
||
`int num_records` (:95-120).
|
||
- Per record (:185-240): `int N[AA,AB,BB]`; `float r_dev[3]`; `float r_mean[3]`; `float theta_dev[3]`;
|
||
`float theta_mean[3]`. Then, by data-block version: v9 = `intensity_threshold` + 14 unused floats; v6-8 = 15 unused;
|
||
v<5 = 2 unused.
|
||
- Then ClusterScore per record (`cluster_separation, total_score (=GenTrain score), original_score, edited`),
|
||
genotype strings, loci names, addresses, and cluster counts (asserted equal to N) (:122-146).
|
||
|
||
### 2.4 GTC reader
|
||
- TOC IDs are listed in idat2gtc.c:862-892 (NUM_SNPS=1 … RAW_X=1000, RAW_Y=1001, GENOTYPES=1002, BASE_CALLS=1003,
|
||
GENOTYPE_SCORES=1004, SCANNER_DATA=1005, CALL_RATE=1006, GENDER=1007, LOGR_DEV=1008, GC10=1009, DX=1010,
|
||
SAMPLE_DATA=1011, B_ALLELE_FREQS=1012, LOGR_RATIOS=1013, PERCENTILES_X/Y=1014/1015, SLIDE_IDENTIFIER=1016).
|
||
- The authoritative spec is `BeadArrayFiles/docs/GTC_File_Format_v5.pdf`.
|
||
- `code2genotype` covers polyploid codes up to octoploid (GenotypeCalls.py:9-57).
|
||
|
||
---
|
||
|
||
## 3. gtc2vcf / idat2gtc (freeseek) — deep dive
|
||
|
||
**License confirmed MIT** (`gtc2vcf/LICENSE`: "Copyright (C) 2018-2025 Giulio Genovese"; per-file MIT header at
|
||
idat2gtc.c:1-25; `nearest_neighbor.c` also MIT). Version string `IDAT2GTC_VERSION "2026-01-26"` (idat2gtc.c:243).
|
||
|
||
### 3.1 What it says it matches (idat2gtc.c:32-106)
|
||
- It replaces AutoConvert 1.6.3.1, AutoConvert 2.0.1.179, IAAP CLI 1.1 and ACLI 2.1.
|
||
- **GenTrain 2.0** = Normalization 1.1.2 + Clustering 6.3.1 + Genotyping 6.3.0.
|
||
**GenTrain 3.0** = Normalization 1.2.0 + Clustering 7.0.0 + Genotyping 7.0.0 (:53-61).
|
||
- Illumina class names cited: `Normalization10(_Dragonfish)`, `Normalization111(_Dragonfish)`, `NormalizationDragonfish`,
|
||
`GenTrain62`, `AutoCallPollerThread`. "Dragonfish" is the Illumina code name for the 1.2.0 normalization.
|
||
- Differences 1.1.2 → 1.2.0 (:76-81):
|
||
- updated MATLAB `madsigma`;
|
||
- HandleScale no longer uses missing loci for bins < 192;
|
||
- new single-channel Infinium-I normalization for bins < 192 loci.
|
||
- Gender logic differs per Illumina tool (:83-93).
|
||
- Deliberately reproduced Illumina bugs (:95-106):
|
||
- `robustfit0` adjfactor is always ≈ 100;
|
||
- normalization-id overflow past 255;
|
||
- zero-valued missing probes are still used in normalization;
|
||
- `trimmean` runs on artificially zeroed values;
|
||
- GenTrain 2.0 counts failed loci as AA in small bins.
|
||
- Presets (idat2gtc.c:4086-4125): 1 = AutoConvert (GT2, GTC v3, gender v1); 2 = AutoConvert 2.0 (GT3, GTC v5,
|
||
gender v2 with autosomal CR threshold 0.97); 3 = IAAP (GT3); 4 = ACLI (default).
|
||
- Defaults: GenCall cutoff 0.15 (:3997); gender `min_loci=100, max_loci=10000, min_x_loci=20, min_y_loci=20,
|
||
call_rate_threshold=0.0 (changed from 0.97), y_threshold=0.3, x_threshold=0.9, x_het_rate_threshold=0.1`
|
||
(:4007-4014).
|
||
- Concordance claim (README.md:233, 236-244, 428): the output is "equivalent to the output of the Illumina GenCall
|
||
algorithm". It gives "same md5sum" with deterministic patched binaries. AutoConvert 2.0, IAAP-CLI and ACLI
|
||
(normalization 1.2.0) "seem to produce the exact same results". AutoConvert (1.1.2) gives "somewhat different
|
||
results". No numeric genotype-concordance table is published, because the claim is bit-exactness.
|
||
**The reference product is not mentioned anywhere in the source.** Whether the reference product's GTC equals ACLI's
|
||
is an open question for SimurgArray to test.
|
||
|
||
### 3.2 IDAT → raw X/Y (fill_array, idat2gtc.c:3711-3745)
|
||
- Infinium II (assay 0): **X = Red mean at address_a, Y = Grn mean at address_a**.
|
||
- Infinium I A/T (assay 1): X = Red[address_a], Y = Red[address_b].
|
||
- Infinium I G/C (assay 2): X = Grn[address_a], Y = Grn[address_b].
|
||
- A locus is used only if **nbeads ≥ 2** in both channels or probes. Otherwise raw = 0 (missing).
|
||
- The IDAT "mean" is already a bead-level trimmed mean computed by the scanner. `trimmed_mean` points at `mean`
|
||
(:578, :635), so no bead-level summarization is needed.
|
||
|
||
### 3.3 Normalization estimation, per bin (normalize, :2890-2945)
|
||
Loci are grouped by `norm_id` (sorted). A bin with < 10 loci is an error (:2923).
|
||
- If GenTrain 3 **and** `norm_id ≥ 100` (Infinium I) **and** bin size < 192 → `normalize_single_bin_single_channel`
|
||
(:2927-2928).
|
||
- Otherwise → `normalize_single_bin` (:2766-2782). Constants: `SAMPLING=400` (:2352), `ROBUST_THRESHOLD=192`
|
||
(:2353-2354).
|
||
|
||
`normalize_single_bin` runs these steps in order:
|
||
1. **remove_outliers** (:2365-2427, only if n ≥ 400). Compute cutoffs for x, y and t = y/(x+y). Low cutoff =
|
||
min(5th smallest, 1st percentile). High cutoff = max(5th largest, 99th percentile). Drop points with any value
|
||
≤ low or ≥ high.
|
||
2. **remove_offset** (:2436-2502; if n < 192, offset = 0):
|
||
- Shift x and y by their minima.
|
||
- x-sweep: 400 points `linspace(0, max x)` on the x-axis. Take the closest data point to each → candidate AA.
|
||
Do a y-sweep the same way → candidate BB. Keep unique points.
|
||
- Fit robust lines (bisquare, tuning 4.685, with intercept): `y = baa + maa·x` on AA and `x = bbb + mbb·y` on BB
|
||
(`matlab_robustfit1`).
|
||
- Intersection: `ox = (bbb + mbb·baa)/(1 − mbb·maa)`, `oy = baa + maa·ox`. Then `offset = (ox + min x, oy + min y)`
|
||
and subtract it (:2482-2491).
|
||
3. **get_nn12_rr12** (:2730-2764): QC fraction of points below 20% of the 95th percentile in both channels, and their
|
||
relative R. These are stored in the XForm only.
|
||
4. **handle_rotation** (:2509-2559; if n < 192, theta = 0):
|
||
- Do a new x-sweep on the offset-corrected data and union it with the prior AA set.
|
||
- Fit a through-origin robust line on AA (`robustfit0`), slope m.
|
||
- `theta = atan(m)`. Rotate: `x' = cos·x + sin·y`, `y' = −sin·x + cos·y`.
|
||
5. **handle_shear** (:2565-2600; if n < 192, shear = 0): do a y-sweep, union with BB, fit `x = m·y` through the
|
||
origin, then `shear = tan(atan(m)) = m` and `x -= shear·y`.
|
||
6. **handle_scale** (:2683-2724), if n ≥ 192:
|
||
- AA = points with `x > 5y` and `x > 0.1·P99(x)`. BB is defined symmetrically.
|
||
- `scale_x = trimmean50(x_AA)` and `scale_y = trimmean50(y_BB)`.
|
||
- Bins < 192 fall back to `base_handle_scale` (:2606-2676): AA = θ° < 10, BB = θ° > 80,
|
||
`scale = trimmean20`. That function's ≥192 branch uses `0.5·trimmean50 + 0.7413·IQR`.
|
||
7. Stored diagnostics: `taa = theta·180/π` and `tbb = (atan(shear) − theta)·180/π` (:2780-2781).
|
||
|
||
`normalize_single_bin_single_channel` (GT3 Infinium I, < 192 loci) is a two-parameter model (:2849-2887):
|
||
- Mirror so x ≥ y.
|
||
- `ythrsh = P2(y)`. Subtract it from both channels.
|
||
- Convert to polar. AA = t < 0.1.
|
||
- `ymean = trimmean20(y_AA) − 50`.
|
||
- `rmean = trimmean20(r, with NaNs compacted and padded with zeros — bug replicated) − 2·ymean`.
|
||
- Result: `offset_x = offset_y = ythrsh + ymean`, `theta = shear = 0`, `scale_x = scale_y = rmean`.
|
||
|
||
Supporting numerics (bit-exactness depends on these):
|
||
- `percentile()` is Illumina's own interpolation, not numpy's default (:2206-2227).
|
||
- `trimmean` uses percentile bounds (:2240-2254).
|
||
- `madsigma_old` vs `madsigma_new` (:1861-1904).
|
||
- `robustfit0` with constant adjfactor ≈ 100 (:1906-1942).
|
||
- `closest_points`: brute force below 2000 points, else Illumina's binned `findClosestSitesToPointsAlongAxis`
|
||
(:2012-2197; also `nearest_neighbor.c`). The binned search can return different neighbours than an exact NN.
|
||
It must be replicated for exactness.
|
||
|
||
### 3.4 Application of the transform and polar coordinates (:2990-3016)
|
||
```
|
||
tx=raw_x-offset_x; ty=raw_y-offset_y
|
||
tx2=cosθ·tx+sinθ·ty; ty2=-sinθ·tx+cosθ·ty; tx3=tx2-shear·ty2
|
||
norm_x = tx3<0 ? 0 : tx3/scale_x ; norm_y = ty2<0 ? 0 : ty2/scale_y
|
||
Theta = atan2(norm_y,norm_x)·2/π (NaN if both 0)
|
||
R = |norm_x|+|norm_y| (FLT_MIN·FLT_EPSILON if both <0)
|
||
```
|
||
This is the same math as BeadArrayFiles. Note that cos θ and sin θ are precomputed in **float** (:3818-3822).
|
||
Picard computes Theta with `atan(y/x)` instead (InfiniumGTCFile.java:193) and its transform in double
|
||
(:254-270). Tiny floating-point differences are therefore expected between implementations.
|
||
|
||
### 3.5 GenCall calling and score (make_calls, :3337-3405; compute_score_call_prelim, :3162-3292)
|
||
- r_dev is floored at 0.1 before calling (:3368-3375).
|
||
- If `R < intensity_threshold` (EGT v9 field), the result is no-call with NaN score (:3163-3166).
|
||
- **gen_std_flair** (:3022-3074) inflates or deflates the cluster SDs:
|
||
- z1 = ½(θAB−θAA)/(σAB+σAA), z2 = ½(θBB−θAB)/(σBB+σAB), mz = min(z1, z2).
|
||
- η = max(mz/3, 1)·min(mz/3, 1). All θ-SDs are multiplied by η and floored at 0.02.
|
||
- r_dev is floored at max(0.2, median of the θ-SDs, r_mean/7).
|
||
- **modilik** (:3077-3159): pick the two nearest clusters by `100·|Δθ| + |ΔR|`. Use half-flat ("vleft/vright") normal
|
||
pdfs in θ and R, depending on cluster ordering. Otherwise use a full normal pdf product.
|
||
- Priors `P_k = (N_k/N + 1)/4`. Posteriors are normalized. The call is the arg-max (:3190-3213).
|
||
- Score components:
|
||
- `score_ap = smf(log10(max/second), 0, 2)`;
|
||
- `score_r1 = smf(R, 0, 0.1)`;
|
||
- per-cluster `score_t` (zmf/smf at 2σ..6σ of θ);
|
||
- `score_r2 = smf(R, 0, r_mean/10)`;
|
||
- `score_r3 = smf(R, r_mean−6σ, r_mean−2σ)`;
|
||
- `score_r4 = zmf(R, r_mean+6σ, r_mean+12σ)`;
|
||
- `score_misclust` = 0.7 (one cluster, not AB), 0.95 (two clusters) or 1.
|
||
- `prelim = score_ap · GenTrainScore(total_score) · score_t · score_r · score_misclust` (:3286-3287).
|
||
- **Final score** `GC = prelim^0.35 · gbellmf(prelim; a=0.504, b=1.71, c=1.08)` (:3294-3302). A NaN score becomes 0.
|
||
A score below the cutoff (0.15) becomes a no-call. `intensity_only` loci are always no-call.
|
||
- Call rate = calls/(num_snps − intensity_only) (:3403-3404).
|
||
- There is **no DNA-score penalization** in this implementation, despite the 2004 tech-note text quoted at :3149-3151.
|
||
|
||
### 3.6 Gender (estimate_gender, :3416-3482)
|
||
Only loci with a non-zero GenTrain score count.
|
||
- **v1:** if more than 20 X loci are called, female if X het rate > 0.1, else male.
|
||
- **v2:** requires autosomal call rate > threshold. Then the result is M if median R(Y) > 0.3; otherwise F if there are
|
||
fewer than 20 X loci; otherwise F if median R(X) > 0.9, else U.
|
||
|
||
### 3.7 BAF and LRR (get_baf_lrr, :3489-3507; calculate_baf_lrr, :3511-3548)
|
||
Following Peiffer 2006:
|
||
```
|
||
if θ == θAB: BAF=0.5, Rexp=R_AB
|
||
if θ < θAB: BAF = 0.5·(θ-θAA)/(θAB-θAA); Rexp = R_AA + (θ-θAA)(R_AA-R_AB)/(θAA-θAB)
|
||
if θ > θAB: BAF = 0.5 + 0.5·(θ-θAB)/(θBB-θAB); Rexp = R_AB + (θ-θAB)(R_BB-R_AB)/(θBB-θAB)
|
||
LRR = log2(R / Rexp) (for intensity-only loci Rexp = cluster r_mean; R==0 → −FLT_MAX)
|
||
BAF clamped to [0,1]
|
||
```
|
||
- **LogRDev** = SD of LRR over autosomal, non-intensity-only, finite loci (:3533-3547).
|
||
- GC10/GC50 = 10th and 50th percentile of GenCall scores above the cutoff (:3592-3604).
|
||
- Intensity percentiles 5/50/95 of raw X and Y (:3552-3570).
|
||
- gtc2vcf's `--adjust-clusters` (gtc2vcf.c:2008-2046) recenters the (θ, R) cluster means using the batch genotypes, with
|
||
a shrinkage prior of weight 0.2 toward the EGT means. It gives less noisy LRR/BAF for large batches, similar in
|
||
spirit to reclustering.
|
||
|
||
### 3.8 BAFregress (`gtc2vcf/BAFregress.c`, MIT)
|
||
It reimplements Jun et al. 2012 BAF regression on homozygous calls against population allele frequency, to estimate
|
||
contamination. Picard also wraps its output (`CreateBafRegressMetricsFile.java`).
|
||
|
||
---
|
||
|
||
## 4. Other projects — notes
|
||
|
||
- **GTCtoVCF (Apache-2.0, archived).**
|
||
- GenCall score in VCF is Phred: `round(-10·log10(min(1, max(1e-5, 1 − GS))))` (GencallFormat.py:62).
|
||
- Groups manifest records by position to "squash" duplicates. Resolves indel REF/ALT by aligning the source-sequence
|
||
flanks to the reference FASTA (`LocusEntryFactory.py`, `BPMRecord.py`).
|
||
- Changelog 1.2.1 added the norm-id byte-wrap fix for Omni5.
|
||
- **Picard `picard.arrays` (MIT).**
|
||
- `InfiniumGTCFile` computes normalized X/Y and Theta (atan-based) (:188-270).
|
||
- `GtcToVcf.polarToEuclidean` (:434-470) converts EGT (θ, R) mean and dev into X/Y mean and dev with delta-method
|
||
variances. This is useful if SimurgArray must emit cluster X/Y in the VCF header or INFO, as the reference product and Picard do.
|
||
- `CompareGtcFiles` is a ready-made GTC diff harness idea.
|
||
- `BpmToNormalizationManifestCsv` exports norm ids.
|
||
- `CollectArraysVariantCallingMetrics` gives array QC metrics used by WARP.
|
||
- **mocha (MIT).** Post-hoc BAF adjustment by cluster medians (`adjust[3][3]`, mocha.c:120). LRR GC regression
|
||
`adjust_lrr` (:384). Median LRR per chromosome, and an HMM for mosaic CNAs. It is a good algorithm reference for a
|
||
reference-product-like CNV module.
|
||
- **PennCNV (Unlicense — public domain).**
|
||
- `detect_cnv.pl` plus `kext/khmm.c`: a 6-state HMM on LRR and BAF with population B-allele frequency (PFB) and GC
|
||
wave adjustment (`cal_gc_snp.pl`, `genomic_wave.pl`).
|
||
- `split_illumina_report.pl` shows the expected GenomeStudio columns ("Log R Ratio", "B Allele Freq").
|
||
- It is the safest CNV code to reuse directly and a natural oracle for CNV calls.
|
||
- **illuminaio (GPL-2).**
|
||
- `R/readIDAT_nonenc.R` handles the binary IDAT v3 format.
|
||
- `R/readIDAT_enc.R` with `src/des.c` decrypts legacy encrypted XML IDATs (DES).
|
||
- Do not link it. SimurgArray must re-derive encrypted-IDAT support clean if needed. Most Infinium genotyping IDATs are
|
||
non-encrypted v3. gtc2vcf's own IDAT reader (MIT, idat2gtc.c:434-760) is the permissive reference.
|
||
- **zCall** has no licence, so treat it as all rights reserved; the idea is described in Goldstein 2012. **optiCall,
|
||
Illuminus, GenoSNP, CRLMM, M3, iCall** are alternative callers; use them only as literature on the normalization.
|
||
Teo 2007 and 2011 document the 400-homozygote sweep.
|
||
- **verifyIDintensity (GPL-3):** a mixture-model contamination estimate from normalized intensities. Re-derive it from
|
||
Jun et al. 2012 or run it as an external oracle. Broad WARP runs it in production.
|
||
- **PGx.**
|
||
- The reference product does star-allele calling from its SNV VCF (plus CNV for CYP2D6).
|
||
- Open options for array VCFs: **PharmCAT** (MPL-2.0; file-level copyleft only, so it can be used as an unmodified
|
||
dependency or subprocess) and **PyPGx** (MIT; explicitly supports chip data). Both are benchmarked on GSA arrays
|
||
(Clin Pharmacol Ther 2025 benchmark).
|
||
- Aldy (IU non-commercial) and Cyrius (PolyForm Strict, WGS-only) are unusable for a product. Stargazer is legacy and
|
||
non-commercial.
|
||
- **Strand / harmonization.**
|
||
- bcftools `+fixref` (MIT) is the default choice.
|
||
- Rayner strand files are a data oracle for TOP/source→plus-strand checks.
|
||
- GenotypeHarmonizer (GPL-3) and plink2 (GPL-3) should be used as subprocesses only.
|
||
- conform-gt is Apache-2.0 per packagers.
|
||
- **Sample QC.** somalier and peddy (MIT) for relatedness, sex and ancestry from VCF. KING and plink2 as oracles.
|
||
|
||
---
|
||
|
||
## 5. Reuse recommendations for SimurgArray
|
||
|
||
| SimurgArray component | Primary reference | Oracle | Direct dependency OK? |
|
||
|---|---|---|---|
|
||
| IDAT v3 parser | gtc2vcf idat reader (MIT); BeadArrayFiles (BSD-2) | illuminaio (run in R), gtc2vcf `--idat` CSV dump | Yes (MIT/BSD) |
|
||
| BPM / EGT / GTC readers | BeadArrayFiles, Picard, gtc2vcf | BeadArrayFiles | Yes |
|
||
| GTC v5 writer | gtc2vcf `gtc_write` (:1001-1152) + GTC_File_Format_v5.pdf | Round-trip via BeadArrayFiles; md5 vs ACLI / the reference product | Yes |
|
||
| Normalization 1.2.0 | Patent US7035740 (expired), Illumina 2006 normalization note, GenTrain 3.0 tech note; gtc2vcf for bug-compatibility | Compare the 13-float XForm per bin with the reference product's GTC `NORMALIZATION_TRANSFORMS` (TOC 400) | Algorithm yes; copying code is OK under MIT with notice |
|
||
| GenCall scoring | Kermani patents, Shen 2005; gtc2vcf | Reference-product GTC GENOTYPE_SCORES (1004) | same |
|
||
| GenTrain clustering (EGT creation) | **No OSS exists.** Use the GenTrain 2.0/3.0 tech notes and US7467117 | GenomeStudio (the reference product does not cluster) | Must be built |
|
||
| BAF/LRR/LogRDev | Peiffer 2006; gtc2vcf | GTC 1012/1013/1008 | Yes |
|
||
| Gender | gtc2vcf v2 logic | GTC 1007 | Yes |
|
||
| GTC→VCF | GTCtoVCF (Apache), Picard GtcToVcf (MIT), gtc2vcf | Reference-product SNV VCF | Yes |
|
||
| CNV | PennCNV (public domain), mocha (MIT) | PennCNV run; reference-product CNV VCF | Yes |
|
||
| Contamination | BAFregress (MIT) | verifyIDintensity (GPL, subprocess) | BAFregress yes |
|
||
| PGx star alleles | PharmCAT (MPL-2.0), PyPGx (MIT) | reference-product PGx output | PharmCAT as unmodified dep; PyPGx yes |
|
||
|
||
### 5.1 License-compatibility summary
|
||
- **Permissive, safe to vendor with notices:** MIT (gtc2vcf, mocha, Picard, PyPGx, somalier, peddy, methylprep, sesame,
|
||
illumina_array_reader), BSD-2 (BeadArrayFiles), BSD-3 (WARP), Apache-2.0 (GTCtoVCF; keep NOTICE),
|
||
Unlicense (PennCNV).
|
||
- **Weak copyleft:** MPL-2.0 (PharmCAT). Modified PharmCAT files must stay open; linking or bundling otherwise is OK.
|
||
Artistic-2.0 (crlmm, minfi).
|
||
- **Strong copyleft (subprocess/oracle only):** GPL-2 illuminaio; GPL-3 plink2, verifyIDintensity, GenotypeHarmonizer,
|
||
akt, ASCAT; GADA (GPL).
|
||
- **Non-commercial or proprietary (avoid):** argyle (CC BY-NC-SA), Aldy (IU), Cyrius (PolyForm Strict), Stargazer,
|
||
QuantiSNP, iPattern, GenoSNP, optiCall (Sanger terms), zCall (no licence), EnsembleCNV (no licence visible).
|
||
|
||
---
|
||
|
||
## 6. Clean-room caveat and suggested process
|
||
|
||
> **Superseded (plan §2):** SimurgArray may reuse MIT/BSD/Apache code with attribution, so no strict clean room is
|
||
> required, and black-box runs of the vendor binary are not part of the plan. The oracles are published
|
||
> vendor-produced GTCs (1000G Broad Omni2.5 set), gtc2vcf and BeadArrayFiles. Kept for context.
|
||
|
||
Illumina's normalization and GenCall algorithms are described in the expired patents (US7035740, US7467117) and in
|
||
public tech notes. The *bit-exact details* (percentile interpolation, the robustfit adjfactor bug, the binned
|
||
closest-point search, the ≥ 2 beads rule, norm-id byte wrap, FLT_MIN·FLT_EPSILON sentinels) are known mainly through
|
||
gtc2vcf. gtc2vcf in turn states that it ports Illumina's GenCall C# source and AutoConvert behaviour.
|
||
|
||
MIT permits reuse, so there is no copyright barrier to reading gtc2vcf. If SimurgArray nonetheless wants a documented
|
||
clean room (for example, to avoid any claim that it derives from Illumina's GenCall source), do this:
|
||
1. Have a "dirty" analyst write a behavioural spec from patents, tech notes and black-box experiments on
|
||
the reference product / IAAP output (GTC diffs).
|
||
2. Have implementers code from that spec only.
|
||
3. Use gtc2vcf only as a black-box **test oracle**: compare GTC TOC fields 400/1002/1004/1007/1008/1012/1013 per
|
||
sample. Do not use it as source material for implementers.
|
||
|
||
Either way, validation must include a direct comparison against **vendor-produced** GTC output (now: the published 1000G Broad GTCs). gtc2vcf's equivalence
|
||
claims stop at ACLI v2.1, and the reference product is not referenced in its code.
|
||
|
||
---
|
||
|
||
## 7. Sources
|
||
- https://github.com/freeseek/gtc2vcf (README.md, idat2gtc.c, gtc2vcf.c, BAFregress.c, LICENSE)
|
||
- https://github.com/freeseek/mocha
|
||
- https://github.com/Illumina/BeadArrayFiles (README.md license section; module/*.py; docs/GTC_File_Format_v5.pdf)
|
||
- https://github.com/Illumina/GTCtoVCF (LICENSE Apache-2.0; README "Semi-archived")
|
||
- https://github.com/broadinstitute/picard (src/main/java/picard/arrays)
|
||
- https://github.com/broadinstitute/warp ; https://broadinstitute.github.io/warp/docs/Pipelines/Illumina_Genotyping_Arrays_Pipeline/README
|
||
- https://github.com/HenrikBengtsson/illuminaio (DESCRIPTION: GPL-2)
|
||
- https://github.com/WGLab/PennCNV (UNLICENSE)
|
||
- https://github.com/jigold/zCall ; https://opticall.bitbucket.io/ ; https://github.com/wtsi-npg/Illuminus ; https://innovation.ox.ac.uk/licence-details/genosnp/
|
||
- https://github.com/HaoKeLab/ensembleCNV ; https://github.com/VanLoo-lab/ascat
|
||
- https://github.com/gjun/verifyIDintensity (GPL-3)
|
||
- https://github.com/PharmGKB/PharmCAT (MPL-2.0) ; https://github.com/sbslee/pypgx (MIT) ; https://github.com/0xTCG/aldy (IU licence) ; https://github.com/Illumina/Cyrius (PolyForm Strict)
|
||
- https://pmc.ncbi.nlm.nih.gov/articles/PMC13339515/ (benchmark of star-allele callers on arrays)
|
||
- https://github.com/chrchang/plink-ng (GPL-3) ; https://github.com/brentp/somalier ; https://github.com/brentp/peddy
|
||
- https://www.chg.ox.ac.uk/~wrayner/strand/ ; https://faculty.washington.edu/browning/conform-gt.html ; https://github.com/biocore-ntnu/snpflip
|
||
- https://github.com/a-xavier/illumina_array_reader ; https://github.com/oheil/IlluminaIdatFiles.jl ; https://github.com/FoxoTech/methylprep ; https://github.com/brj0/mepylome ; https://github.com/molgenis/AGCT
|
||
- [ref-docs:output-files] (reference-product GTC/VCF outputs)
|