kemal/SimurgArray
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
cc8c9341679d2564ab718dc2278caa97c309a241
SimurgArray/docs/research/R2_open_source_ecosystem.md
T
kemal 854cf95029 Add sanitized Phase-0 research archive under docs/research 
Research notes R1-R4, documentation-study files A-E (source of the
functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use
neutral naming and current project decisions (plan §2).
2026-09-27 16:27:34 +01:00

37 KiB
Raw Blame History

R2 — Open-source ecosystem for Illumina Infinium genotyping-array analysis

Archived Phase-0 study (sanitized copy, see README.md). Written 2026-09-27, before the decisions in SIMURGARRAY_MASTER_PLAN.md §2. The analysed vendor product is called "the reference product" and its CLI refcli; its documentation URLs are replaced by [ref-docs:…] tokens. Where this file speaks of a clean room, legal clearance, or running/benchmarking the vendor binary, plan §2 supersedes it: personal research; MIT/BSD/Apache code may be reused with attribution; GPL tools are subprocess oracles only; vendor binaries are never run for comparison, inspected or decompiled; oracles are published vendor-produced GTCs, gtc2vcf and BeadArrayFiles.

Survey date: 2026-09-27. Scope: projects relevant to building SimurgArray, an independent reimplementation of the reference product (IDAT → normalization → GenCall → GTC → VCF, BAF/LRR, CNV, sex, contamination, PGx star alleles).

Local shallow clones used for this report live in <repo>/refs/: BeadArrayFiles, GTCtoVCF, gtc2vcf, mocha, illuminaio, picard (sparse: src/main/java/picard/arrays), PennCNV, zCall, verifyIDintensity, AGCT. All file:line references below point at those clones. Commit dates are from git log -1 on the clones, or from the GitHub API (pushed_at) where noted.

0. Key findings (TL;DR)

  1. freeseek/gtc2vcf idat2gtc.c (MIT, C, active: last commit 2026-02-26) is a complete open reimplementation of Illumina AutoCall/GenCall for an existing cluster file. It covers IDAT parsing, per-bin affine normalization (versions 1.1.2 = GenTrain 2.0 and 1.2.0 = GenTrain 3.0), the GenCall Bayesian/fuzzy call and score, gender estimation, BAF/LRR, LogR dev, GC10/GC50 and writing GTC v3/v5. The author says the output is "equivalent to the output of the Illumina GenCall algorithm". With --autocall-date "" and a --preset (1 = AutoConvert 1.6.3.1, 2 = AutoConvert 2.0.1.179, 3 = IAAP CLI, 4 = ACLI) he says you "should be able to verify that you get the same md5sum" as patched, deterministic Illumina binaries (README.md:233-244). This is byte-identical GTC, a stronger claim than genotype concordance. It does not implement GenTrain clustering, so it cannot create an EGT.
    • Provenance note for clean-room purposes: the header (idat2gtc.c:27-30, 63-67) says it "reimplements functionalities and ideas present in AutoConvert (v1.6.3.1), GTCtoVCF, BeadArrayFiles" and cites Illumina's GenCall C# source code (NormalizationInfinium.cs, GenTrain60.cs, Utils.cs). It says these were "provided by Illumina without license restrictions". It also deliberately re-implements Illumina bugs for bit-exactness (idat2gtc.c:95-106). It is MIT-licensed, so it is legally reusable with attribution. However, it is effectively a port of Illumina code. If SimurgArray needs a strict clean room, route it through a spec-writer/implementer split (see §6). (Not required: plan §2 allows MIT reuse with attribution.)
    • The key patents have expired: US7035740 (normalization) on 2024-05-05 and US7467117 (clustering/genotyping) on 2024-03-24 (idat2gtc.c:47-51).
  2. Illumina/BeadArrayFiles is NOT Apache-2.0. The README carries a BSD-2-Clause ("FreeBSD") license, Copyright (c) 2020 Illumina (README.md:84-111). The repo has no LICENSE file, so the GitHub API reports no license. Last commit 2025-01-30. It is the reference parser for GTC/BPM/EGT and holds the canonical normalization application (NormalizationTransform.normalize_intensities). It does not hold the estimation of the transform.
  3. Illumina/GTCtoVCF is Apache-2.0 and ARCHIVED (GitHub archived: true). Its README says "Semi-archived … As of May 2023, it is now recommended to use … [product]". It only supports Python 2.7. It is still useful as an algorithm reference for indel/reference-allele resolution from manifest source sequences, and for the VCF FORMAT layout (GT, GS as Phred, BAF, LRR).
  4. broadinstitute/picard picard.arrays (MIT, Java, active) is a second, independent, permissively licensed set of GTC/BPM/EGT/manifest readers. It also includes GtcToVcf, a normalized-intensity computation, the conversion of cluster (Theta, R) mean/dev to X/Y, and QC metric tools (BafRegress, VerifyIDIntensity, CollectArraysVariantCallingMetrics). It is used in Broad WARP's production Illumina genotyping pipeline (BSD-3-Clause).
  5. No open-source GenTrain clustering (EGT writer) exists. Searches found only readers. SimurgArray will have to build cluster training from the Illumina GenTrain 2/3 technical notes and the patents.
  6. Licensing hazards: illuminaio (GPL-2), plink2 (GPL-3), GenotypeHarmonizer/systemsgenetics (GPL-3), verifyIDintensity (GPL-3), argyle (CC BY-NC-SA 4.0), Aldy (non-commercial IU license), Cyrius (PolyForm Strict, no commercial use), zCall (no license = all rights reserved). Use these only as external executables or test oracles, never as linked code.

1. Summary table

Reuse legend. DEP = safe as a linked dependency. ALG = algorithm reference. ORACLE = run as a test oracle (black box). IGN = ignore.

Project URL License Lang Last activity What it implements Faithfulness to Illumina SimurgArray reuse License risk
gtc2vcf / idat2gtc / BAFregress github.com/freeseek/gtc2vcf MIT (LICENSE; per-file MIT headers) C (bcftools plugins) 2026-02-26 (v2026-01-26) IDAT/BPM/EGT/GTC/CSV parsing; normalization 1.1.2 and 1.2.0; GenCall calling and score; gender; BAF/LRR; GTC writer; GTC→VCF; cluster re-centering; BAFregress contamination Claims md5-identical GTC vs AutoConvert, AutoConvert 2.0, IAAP-CLI and ACLI when timestamps are disabled ALG + ORACLE; DEP is possible (MIT) Low legally. Clean-room provenance caveat (port of Illumina C#)
mocha github.com/freeseek/mocha MIT C 2025-08-22 Mosaic chromosomal alteration HMM on BAF/LRR/phase; BAF cluster median adjustment; LRR GC-wave regression N/A (downstream) ALG (CNV/mCA module) Low
BeadArrayFiles github.com/Illumina/BeadArrayFiles BSD-2-Clause (in README; no LICENSE file) Python 3 2025-01-30 GTC v3-5, BPM v3-5 (locus v4,7,8), EGT v3 (data-block v<5, 6-8, 9) readers; NormalizationTransform; LocusAggregate; report examples Official Illumina DEP / ALG / ORACLE (format spec: docs/GTC_File_Format_v5.pdf) Low (keep notice)
GTCtoVCF github.com/Illumina/GTCtoVCF Apache-2.0 Python 2.7 Archived (last code 2024-03-15) GTC+BPM+CSV+FASTA → VCF; indel ref-allele resolution; GS as Phred; BAF/LRR FORMAT Official, superseded by the reference product ALG (VCF conversion, indel logic); ORACLE Low (NOTICE requirement)
Picard picard.arrays github.com/broadinstitute/picard MIT Java 2026-07-31 InfiniumGTCFile, IlluminaBPMFile, InfiniumEGTFile, normalization manifest, GtcToVcf, extended manifest, BafRegress and VerifyIDIntensity metric wrappers, arrays calling metrics, VcfToAdpc Production use in WARP; reads Illumina-produced GTC (does not call) ALG + ORACLE; DEP if JVM is acceptable Low
WARP Illumina Genotyping Array github.com/broadinstitute/warp BSD-3-Clause WDL active 2026-09 Production pipeline: IAAP-CLI → Picard GtcToVcf → metrics, VerifyIDIntensity, BafRegress, zCall, fingerprinting Uses Illumina binaries ALG (pipeline/QC design), output spec Low
bcftools github.com/samtools/bcftools MIT/Expat (GPL if built with GSL) C 2026-09-21 Host for gtc2vcf plugins; +fixref, norm, +split-vep — DEP (subprocess) or ORACLE Low if built without GSL
illuminaio github.com/HenrikBengtsson/illuminaio (Bioconductor) GPL-2 R + C (DES) 2024-05-12 IDAT reader (non-encrypted binary and encrypted DES/XML IDATs), BPM, BGX Well tested ALG only (clean re-derivation of encrypted IDAT) High if linked
illumina_array_reader github.com/a-xavier/illumina_array_reader MIT Rust 2023 IDAT reader (illuminaio port) → polars — DEP candidate (Rust) Low. It is a port of GPL code, so verify provenance
IlluminaIdatFiles.jl github.com/oheil/IlluminaIdatFiles.jl MIT (per repo) Julia — IDAT reader — IGN / ALG Low
methylprep / methylcheck github.com/FoxoTech/methylprep MIT Python 2023-08 (stale) IDAT parsing (methylation), NOOB, QC Methylation only ALG (IDAT parsing) Low
mepylome github.com/brj0/mepylome MIT Python/C++ 2025 Fast IDAT parsing (methylation), CNV from methylation arrays — ALG Low
minfi bioconductor/hansenlab/minfi Artistic-2.0 R 2024-04 Methylation arrays — IGN Medium
sesame github.com/zwdzwd/sesame MIT + file R 2026-09-23 Methylation arrays, pOOBAH, IDAT via own reader — IGN (methylation) Low
crlmm Bioconductor Artistic-2.0 R/C Bioc release CRLMM genotype calling for Illumina/Affy; CNV Alternative caller; published concordance vs GenCall (Ritchie 2011) ALG (alt. caller) Medium
argyle github.com/andrewparkermorgan/argyle CC BY-NC-SA 4.0 R 2018-03 (dormant) Mouse MUGA arrays; reads GenomeStudio FinalReport; QC; BAF/LRR — IGN / ALG High (non-commercial)
zCall github.com/jigold/zCall none (all rights reserved) Python 2015-02 Rare-variant re-caller using z-score thresholds on normalized X/Y after GenCall Post-GenCall rescue ALG (idea only) High (no license)
optiCall opticall.bitbucket.io (Sanger) Sanger terms (not OSI; binary + source on request) C++ ~2012 Rare/common-variant caller using Illumina-normalized intensities Alt. caller ALG (paper only) High
Illuminus github.com/wtsi-npg/Illuminus unclear/none in repo C stale Mixture-model caller (Teo 2007). Paper documents Illumina normalization (400 homozygotes) Alt. caller ALG (paper) High (unclear)
GenoSNP Oxford Univ Innovation licence Academic licence C stale Within-sample VB caller Alt. caller IGN High
PennCNV github.com/WGLab/PennCNV Unlicense (public domain) (UNLICENSE file) Perl + C (kext HMM) 2024-05-19 CNV HMM on LRR/BAF with PFB and GC model; split_illumina_report.pl, compile_pfb.pl, cal_gc_snp.pl De-facto standard for array CNV DEP / ALG (HMM, PFB, GC-wave model) None
QuantiSNP (Oxford, binary) Academic binary licence MATLAB runtime stale OHMM CNV — IGN High
iPattern (Broad/Harvard) Academic licence R/Perl stale Intensity-pattern CNV — IGN High
EnsembleCNV github.com/HaoKeLab/ensembleCNV none visible R/Perl ~2019-2021 Meta-caller over PennCNV/QuantiSNP/iPattern; CNVR genotyping — ALG (ensemble idea) High (no license)
GADA Bioconductor/CRAN "gada" GPL R/C stale Sparse Bayesian segmentation of LRR — ALG High if linked
ASCAT github.com/VanLoo-lab/ascat GPL-3 (per repo) R 2026-02 Allele-specific CN for tumours from BAF/LRR — ALG High if linked
plink2 github.com/chrchang/plink-ng GPL-3 (2.0/COPYING) C++ 2026-09-27 Genotype QC, sex check, relatedness (KING-robust), PCA — ORACLE (subprocess) High if linked
bcftools +fixref samtools/bcftools MIT C active Strand/REF fixing vs reference — DEP / ORACLE Low
snpflip github.com/biocore-ntnu/snpflip MIT per PyPI (no LICENSE in repo) Python Archived 2019 Detect reverse/ambiguous-strand SNPs — IGN / ALG Low-Med
Will Rayner strand files chg.ox.ac.uk/~wrayner/strand/ Data, no explicit licence data + Perl maintained Per-chip strand/build files (TOP/source strand → + strand) Community standard ORACLE (strand truth) Medium (data terms unclear)
GenotypeHarmonizer molgenis/systemsgenetics GPL-3 Java 2026-03 Strand alignment via LD with reference — ORACLE High if linked
conform-gt faculty.washington.edu/browning Apache-2.0 (reported) Java stale Conform VCF to reference panel (strand/allele) — ORACLE Low
AGCT github.com/molgenis/AGCT (none visible) shell — IDAT→GTC pipeline using IAAP — IGN —
peddy github.com/brentp/peddy MIT Python 2026-02 Pedigree/sex/ancestry QC from VCF — DEP/ALG Low
somalier github.com/brentp/somalier MIT Nim 2026-09-25 Relatedness/sample swap from sites, works on array VCF — DEP/ALG Low
KING kingrelatedness.com Binary, free download; source by request C++ — Kinship — ORACLE Medium
akt github.com/Illumina/akt GPL-3 (per repo) C++ stale Ancestry and kinship tools — ORACLE High if linked
verifyIDintensity github.com/gjun/verifyIDintensity GPL-3 C++ 2015-05 Contamination from array intensities (Jun 2012) Paper method ALG (re-derive); ORACLE High if linked
BAFregress inside gtc2vcf (BAFregress.c) MIT C 2025-08 Contamination via BAF regression on hom-calls (Jun 2012) Same model as original DEP/ALG Low
PharmCAT github.com/PharmGKB/PharmCAT MPL-2.0 Java 2026-09-25 Star-allele matcher from VCF (array or WGS), CPIC phenotypes Benchmarked on GSA/PGx arrays ORACLE / DEP (file-level copyleft only) Medium (MPL is file-level)
PyPGx (Stargazer successor) github.com/sbslee/pypgx MIT Python 2026-08 Star alleles from NGS and chip VCFs (supports array input) Benchmarked on GSA ALG / ORACLE Low
Stargazer stargazer.gs.washington.edu Academic, non-commercial Python legacy Star alleles (NGS) — IGN High
Aldy github.com/0xTCG/aldy IU non-commercial licence Python/C 2026-04 Star alleles (NGS; VCF input) — IGN / ORACLE only High
Cyrius github.com/Illumina/Cyrius PolyForm Strict 1.0.0 Python 2023-10 CYP2D6 from WGS — IGN High
BeadArrayPython3 github.com/dbrami/BeadArrayPython3 (fork of BSD-2) Python — Py3 port of BeadArrayFiles — IGN (upstream is Py3 now) Low

"Parses [product] outputs / 'refcli'": no public GitHub repo was found that parses reference-product-specific outputs (the CNV VCF, PGx JSON/TSV, or refcli CLI logs). The reference product's GTC is v5, and Illumina's docs point to BeadArrayFiles for parsing it. The SNV VCF FORMAT fields (GT, GS, BAF, LRR) are compatible with what gtc2vcf/GTCtoVCF/Picard emit. So these three are the closest analogues for output-schema compatibility tests.

2. BeadArrayFiles (Illumina) — formats and normalization application

License (corrected): BSD-2-Clause text in BeadArrayFiles/README.md:84-111 ("Copyright (c) 2020, Illumina … FreeBSD Project"). It is not Apache-2.0. Apache-2.0 applies to GTCtoVCF (GTCtoVCF/LICENSE).

2.1 NormalizationTransform (GenotypeCalls.py:625-698)

  • Binary layout, 52 bytes per transform (read_normalization_transform, :641-652): int32 version, then float32 offset_x, offset_y, scale_x, scale_y, shear, theta (:637-639). The remaining 6 floats are reserved. In gtc2vcf they are cvx, cvy, nn12, rr12, taa, tbb (idat2gtc.c:902-916).
  • normalize_intensities(x, y, threshold=True) (GenotypeCalls.py:668-698): 
    if x <= 0 and y <= 0: return (nan, nan)
tx  = x - offset_x ;  ty = y - offset_y                # translate
tx2 =  cos(theta)*tx + sin(theta)*ty                    # rotate by -theta
ty2 = -sin(theta)*tx + cos(theta)*ty
tx3 = tx2 - shear*ty2                                   # shear (x only)
xn = tx3/scale_x ; yn = ty2/scale_y                     # scale
if threshold: xn = max(xn,0); yn = max(yn,0)            # clamp at 0
  • rect_to_polar(x, y) (:654-666): R = x + y, theta = atan2(y, x) * 2/pi. It returns (nan, nan) if x == y == 0.
  • get_normalized_intensities(normalization_lookups) (:570-582) applies transform lookups[i] per locus.

2.2 Normalization-bin lookup (BeadPoolManifest.py:90-130)

  • A per-locus normalization_id byte is read (it must be < 100, :91-94). Then norm_id += 100 * assay_type and norm_id %= 256 (:118-124). The mod is commented "To mimic the byte-wrapping behavior from GenomeStudio, AutoCall, IAAP". This affects Omni5 (see GTCtoVCF changelog 1.2.1).
  • Transform index = rank of norm_id among the sorted set of distinct ids (:125-130). The GTC stores transforms in that order.
  • assay_type: 0 = Infinium II, 1 = Infinium I A/T (red), 2 = Infinium I G/C (green). gtc2vcf infers it from probe and source sequences (idat2gtc.c:1242-1272).

2.3 EGT reader (ClusterFile.py:83-240)

  • Header: int version (must be 3), strings gencall_version, cluster_version, call_version, normalization_version, date_created, a byte is_wgt (must be 1), string manifest_name, int data_block_version, string opa, int num_records (:95-120).
  • Per record (:185-240): int N[AA,AB,BB]; float r_dev[3]; float r_mean[3]; float theta_dev[3]; float theta_mean[3]. Then, by data-block version: v9 = intensity_threshold + 14 unused floats; v6-8 = 15 unused; v<5 = 2 unused.
  • Then ClusterScore per record (cluster_separation, total_score (=GenTrain score), original_score, edited), genotype strings, loci names, addresses, and cluster counts (asserted equal to N) (:122-146).

2.4 GTC reader

  • TOC IDs are listed in idat2gtc.c:862-892 (NUM_SNPS=1 … RAW_X=1000, RAW_Y=1001, GENOTYPES=1002, BASE_CALLS=1003, GENOTYPE_SCORES=1004, SCANNER_DATA=1005, CALL_RATE=1006, GENDER=1007, LOGR_DEV=1008, GC10=1009, DX=1010, SAMPLE_DATA=1011, B_ALLELE_FREQS=1012, LOGR_RATIOS=1013, PERCENTILES_X/Y=1014/1015, SLIDE_IDENTIFIER=1016).
  • The authoritative spec is BeadArrayFiles/docs/GTC_File_Format_v5.pdf.
  • code2genotype covers polyploid codes up to octoploid (GenotypeCalls.py:9-57).

3. gtc2vcf / idat2gtc (freeseek) — deep dive

License confirmed MIT (gtc2vcf/LICENSE: "Copyright (C) 2018-2025 Giulio Genovese"; per-file MIT header at idat2gtc.c:1-25; nearest_neighbor.c also MIT). Version string IDAT2GTC_VERSION "2026-01-26" (idat2gtc.c:243).

3.1 What it says it matches (idat2gtc.c:32-106)

  • It replaces AutoConvert 1.6.3.1, AutoConvert 2.0.1.179, IAAP CLI 1.1 and ACLI 2.1.
  • GenTrain 2.0 = Normalization 1.1.2 + Clustering 6.3.1 + Genotyping 6.3.0. GenTrain 3.0 = Normalization 1.2.0 + Clustering 7.0.0 + Genotyping 7.0.0 (:53-61).
  • Illumina class names cited: Normalization10(_Dragonfish), Normalization111(_Dragonfish), NormalizationDragonfish, GenTrain62, AutoCallPollerThread. "Dragonfish" is the Illumina code name for the 1.2.0 normalization.
  • Differences 1.1.2 → 1.2.0 (:76-81):
    • updated MATLAB madsigma;
    • HandleScale no longer uses missing loci for bins < 192;
    • new single-channel Infinium-I normalization for bins < 192 loci.
  • Gender logic differs per Illumina tool (:83-93).
  • Deliberately reproduced Illumina bugs (:95-106):
    • robustfit0 adjfactor is always ≈ 100;
    • normalization-id overflow past 255;
    • zero-valued missing probes are still used in normalization;
    • trimmean runs on artificially zeroed values;
    • GenTrain 2.0 counts failed loci as AA in small bins.
  • Presets (idat2gtc.c:4086-4125): 1 = AutoConvert (GT2, GTC v3, gender v1); 2 = AutoConvert 2.0 (GT3, GTC v5, gender v2 with autosomal CR threshold 0.97); 3 = IAAP (GT3); 4 = ACLI (default).
  • Defaults: GenCall cutoff 0.15 (:3997); gender min_loci=100, max_loci=10000, min_x_loci=20, min_y_loci=20, call_rate_threshold=0.0 (changed from 0.97), y_threshold=0.3, x_threshold=0.9, x_het_rate_threshold=0.1 (:4007-4014).
  • Concordance claim (README.md:233, 236-244, 428): the output is "equivalent to the output of the Illumina GenCall algorithm". It gives "same md5sum" with deterministic patched binaries. AutoConvert 2.0, IAAP-CLI and ACLI (normalization 1.2.0) "seem to produce the exact same results". AutoConvert (1.1.2) gives "somewhat different results". No numeric genotype-concordance table is published, because the claim is bit-exactness. The reference product is not mentioned anywhere in the source. Whether the reference product's GTC equals ACLI's is an open question for SimurgArray to test.

3.2 IDAT → raw X/Y (fill_array, idat2gtc.c:3711-3745)

  • Infinium II (assay 0): X = Red mean at address_a, Y = Grn mean at address_a.
  • Infinium I A/T (assay 1): X = Red[address_a], Y = Red[address_b].
  • Infinium I G/C (assay 2): X = Grn[address_a], Y = Grn[address_b].
  • A locus is used only if nbeads ≥ 2 in both channels or probes. Otherwise raw = 0 (missing).
  • The IDAT "mean" is already a bead-level trimmed mean computed by the scanner. trimmed_mean points at mean (:578, :635), so no bead-level summarization is needed.

3.3 Normalization estimation, per bin (normalize, :2890-2945)

Loci are grouped by norm_id (sorted). A bin with < 10 loci is an error (:2923).

  • If GenTrain 3 and norm_id ≥ 100 (Infinium I) and bin size < 192 → normalize_single_bin_single_channel (:2927-2928).
  • Otherwise → normalize_single_bin (:2766-2782). Constants: SAMPLING=400 (:2352), ROBUST_THRESHOLD=192 (:2353-2354).

normalize_single_bin runs these steps in order:

  1. remove_outliers (:2365-2427, only if n ≥ 400). Compute cutoffs for x, y and t = y/(x+y). Low cutoff = min(5th smallest, 1st percentile). High cutoff = max(5th largest, 99th percentile). Drop points with any value ≤ low or ≥ high.
  2. remove_offset (:2436-2502; if n < 192, offset = 0):
    • Shift x and y by their minima.
    • x-sweep: 400 points linspace(0, max x) on the x-axis. Take the closest data point to each → candidate AA. Do a y-sweep the same way → candidate BB. Keep unique points.
    • Fit robust lines (bisquare, tuning 4.685, with intercept): y = baa + maa·x on AA and x = bbb + mbb·y on BB (matlab_robustfit1).
    • Intersection: ox = (bbb + mbb·baa)/(1 − mbb·maa), oy = baa + maa·ox. Then offset = (ox + min x, oy + min y) and subtract it (:2482-2491).
  3. get_nn12_rr12 (:2730-2764): QC fraction of points below 20% of the 95th percentile in both channels, and their relative R. These are stored in the XForm only.
  4. handle_rotation (:2509-2559; if n < 192, theta = 0):
    • Do a new x-sweep on the offset-corrected data and union it with the prior AA set.
    • Fit a through-origin robust line on AA (robustfit0), slope m.
    • theta = atan(m). Rotate: x' = cos·x + sin·y, y' = −sin·x + cos·y.
  5. handle_shear (:2565-2600; if n < 192, shear = 0): do a y-sweep, union with BB, fit x = m·y through the origin, then shear = tan(atan(m)) = m and x -= shear·y.
  6. handle_scale (:2683-2724), if n ≥ 192:
    • AA = points with x > 5y and x > 0.1·P99(x). BB is defined symmetrically.
    • scale_x = trimmean50(x_AA) and scale_y = trimmean50(y_BB).
    • Bins < 192 fall back to base_handle_scale (:2606-2676): AA = θ° < 10, BB = θ° > 80, scale = trimmean20. That function's ≥192 branch uses 0.5·trimmean50 + 0.7413·IQR.
  7. Stored diagnostics: taa = theta·180/π and tbb = (atan(shear) − theta)·180/π (:2780-2781).

normalize_single_bin_single_channel (GT3 Infinium I, < 192 loci) is a two-parameter model (:2849-2887):

  • Mirror so x ≥ y.
  • ythrsh = P2(y). Subtract it from both channels.
  • Convert to polar. AA = t < 0.1.
  • ymean = trimmean20(y_AA) − 50.
  • rmean = trimmean20(r, with NaNs compacted and padded with zeros — bug replicated) − 2·ymean.
  • Result: offset_x = offset_y = ythrsh + ymean, theta = shear = 0, scale_x = scale_y = rmean.

Supporting numerics (bit-exactness depends on these):

  • percentile() is Illumina's own interpolation, not numpy's default (:2206-2227).
  • trimmean uses percentile bounds (:2240-2254).
  • madsigma_old vs madsigma_new (:1861-1904).
  • robustfit0 with constant adjfactor ≈ 100 (:1906-1942).
  • closest_points: brute force below 2000 points, else Illumina's binned findClosestSitesToPointsAlongAxis (:2012-2197; also nearest_neighbor.c). The binned search can return different neighbours than an exact NN. It must be replicated for exactness.

3.4 Application of the transform and polar coordinates (:2990-3016)

tx=raw_x-offset_x; ty=raw_y-offset_y
tx2=cosθ·tx+sinθ·ty; ty2=-sinθ·tx+cosθ·ty; tx3=tx2-shear·ty2
norm_x = tx3<0 ? 0 : tx3/scale_x ;  norm_y = ty2<0 ? 0 : ty2/scale_y
Theta = atan2(norm_y,norm_x)·2/π     (NaN if both 0)
R     = |norm_x|+|norm_y|            (FLT_MIN·FLT_EPSILON if both <0)

This is the same math as BeadArrayFiles. Note that cos θ and sin θ are precomputed in float (:3818-3822). Picard computes Theta with atan(y/x) instead (InfiniumGTCFile.java:193) and its transform in double (:254-270). Tiny floating-point differences are therefore expected between implementations.

3.5 GenCall calling and score (make_calls, :3337-3405; compute_score_call_prelim, :3162-3292)

  • r_dev is floored at 0.1 before calling (:3368-3375).
  • If R < intensity_threshold (EGT v9 field), the result is no-call with NaN score (:3163-3166).
  • gen_std_flair (:3022-3074) inflates or deflates the cluster SDs:
    • z1 = ½(θAB−θAA)/(σAB+σAA), z2 = ½(θBB−θAB)/(σBB+σAB), mz = min(z1, z2).
    • η = max(mz/3, 1)·min(mz/3, 1). All θ-SDs are multiplied by η and floored at 0.02.
    • r_dev is floored at max(0.2, median of the θ-SDs, r_mean/7).
  • modilik (:3077-3159): pick the two nearest clusters by 100·|Δθ| + |ΔR|. Use half-flat ("vleft/vright") normal pdfs in θ and R, depending on cluster ordering. Otherwise use a full normal pdf product.
  • Priors P_k = (N_k/N + 1)/4. Posteriors are normalized. The call is the arg-max (:3190-3213).
  • Score components:
    • score_ap = smf(log10(max/second), 0, 2);
    • score_r1 = smf(R, 0, 0.1);
    • per-cluster score_t (zmf/smf at 2σ..6σ of θ);
    • score_r2 = smf(R, 0, r_mean/10);
    • score_r3 = smf(R, r_mean−6σ, r_mean−2σ);
    • score_r4 = zmf(R, r_mean+6σ, r_mean+12σ);
    • score_misclust = 0.7 (one cluster, not AB), 0.95 (two clusters) or 1.
  • prelim = score_ap · GenTrainScore(total_score) · score_t · score_r · score_misclust (:3286-3287).
  • Final score GC = prelim^0.35 · gbellmf(prelim; a=0.504, b=1.71, c=1.08) (:3294-3302). A NaN score becomes 0. A score below the cutoff (0.15) becomes a no-call. intensity_only loci are always no-call.
  • Call rate = calls/(num_snps − intensity_only) (:3403-3404).
  • There is no DNA-score penalization in this implementation, despite the 2004 tech-note text quoted at :3149-3151.

3.6 Gender (estimate_gender, :3416-3482)

Only loci with a non-zero GenTrain score count.

  • v1: if more than 20 X loci are called, female if X het rate > 0.1, else male.
  • v2: requires autosomal call rate > threshold. Then the result is M if median R(Y) > 0.3; otherwise F if there are fewer than 20 X loci; otherwise F if median R(X) > 0.9, else U.

3.7 BAF and LRR (get_baf_lrr, :3489-3507; calculate_baf_lrr, :3511-3548)

Following Peiffer 2006:

if θ == θAB:  BAF=0.5, Rexp=R_AB
if θ <  θAB:  BAF = 0.5·(θ-θAA)/(θAB-θAA);        Rexp = R_AA + (θ-θAA)(R_AA-R_AB)/(θAA-θAB)
if θ >  θAB:  BAF = 0.5 + 0.5·(θ-θAB)/(θBB-θAB);  Rexp = R_AB + (θ-θAB)(R_BB-R_AB)/(θBB-θAB)
LRR = log2(R / Rexp)   (for intensity-only loci Rexp = cluster r_mean; R==0 → −FLT_MAX)
BAF clamped to [0,1]
  • LogRDev = SD of LRR over autosomal, non-intensity-only, finite loci (:3533-3547).
  • GC10/GC50 = 10th and 50th percentile of GenCall scores above the cutoff (:3592-3604).
  • Intensity percentiles 5/50/95 of raw X and Y (:3552-3570).
  • gtc2vcf's --adjust-clusters (gtc2vcf.c:2008-2046) recenters the (θ, R) cluster means using the batch genotypes, with a shrinkage prior of weight 0.2 toward the EGT means. It gives less noisy LRR/BAF for large batches, similar in spirit to reclustering.

3.8 BAFregress (gtc2vcf/BAFregress.c, MIT)

It reimplements Jun et al. 2012 BAF regression on homozygous calls against population allele frequency, to estimate contamination. Picard also wraps its output (CreateBafRegressMetricsFile.java).

4. Other projects — notes

  • GTCtoVCF (Apache-2.0, archived).
    • GenCall score in VCF is Phred: round(-10·log10(min(1, max(1e-5, 1 − GS)))) (GencallFormat.py:62).
    • Groups manifest records by position to "squash" duplicates. Resolves indel REF/ALT by aligning the source-sequence flanks to the reference FASTA (LocusEntryFactory.py, BPMRecord.py).
    • Changelog 1.2.1 added the norm-id byte-wrap fix for Omni5.
  • Picard picard.arrays (MIT).
    • InfiniumGTCFile computes normalized X/Y and Theta (atan-based) (:188-270).
    • GtcToVcf.polarToEuclidean (:434-470) converts EGT (θ, R) mean and dev into X/Y mean and dev with delta-method variances. This is useful if SimurgArray must emit cluster X/Y in the VCF header or INFO, as the reference product and Picard do.
    • CompareGtcFiles is a ready-made GTC diff harness idea.
    • BpmToNormalizationManifestCsv exports norm ids.
    • CollectArraysVariantCallingMetrics gives array QC metrics used by WARP.
  • mocha (MIT). Post-hoc BAF adjustment by cluster medians (adjust[3][3], mocha.c:120). LRR GC regression adjust_lrr (:384). Median LRR per chromosome, and an HMM for mosaic CNAs. It is a good algorithm reference for a reference-product-like CNV module.
  • PennCNV (Unlicense — public domain).
    • detect_cnv.pl plus kext/khmm.c: a 6-state HMM on LRR and BAF with population B-allele frequency (PFB) and GC wave adjustment (cal_gc_snp.pl, genomic_wave.pl).
    • split_illumina_report.pl shows the expected GenomeStudio columns ("Log R Ratio", "B Allele Freq").
    • It is the safest CNV code to reuse directly and a natural oracle for CNV calls.
  • illuminaio (GPL-2).
    • R/readIDAT_nonenc.R handles the binary IDAT v3 format.
    • R/readIDAT_enc.R with src/des.c decrypts legacy encrypted XML IDATs (DES).
    • Do not link it. SimurgArray must re-derive encrypted-IDAT support clean if needed. Most Infinium genotyping IDATs are non-encrypted v3. gtc2vcf's own IDAT reader (MIT, idat2gtc.c:434-760) is the permissive reference.
  • zCall has no licence, so treat it as all rights reserved; the idea is described in Goldstein 2012. optiCall, Illuminus, GenoSNP, CRLMM, M3, iCall are alternative callers; use them only as literature on the normalization. Teo 2007 and 2011 document the 400-homozygote sweep.
  • verifyIDintensity (GPL-3): a mixture-model contamination estimate from normalized intensities. Re-derive it from Jun et al. 2012 or run it as an external oracle. Broad WARP runs it in production.
  • PGx.
    • The reference product does star-allele calling from its SNV VCF (plus CNV for CYP2D6).
    • Open options for array VCFs: PharmCAT (MPL-2.0; file-level copyleft only, so it can be used as an unmodified dependency or subprocess) and PyPGx (MIT; explicitly supports chip data). Both are benchmarked on GSA arrays (Clin Pharmacol Ther 2025 benchmark).
    • Aldy (IU non-commercial) and Cyrius (PolyForm Strict, WGS-only) are unusable for a product. Stargazer is legacy and non-commercial.
  • Strand / harmonization.
    • bcftools +fixref (MIT) is the default choice.
    • Rayner strand files are a data oracle for TOP/source→plus-strand checks.
    • GenotypeHarmonizer (GPL-3) and plink2 (GPL-3) should be used as subprocesses only.
    • conform-gt is Apache-2.0 per packagers.
  • Sample QC. somalier and peddy (MIT) for relatedness, sex and ancestry from VCF. KING and plink2 as oracles.

5. Reuse recommendations for SimurgArray

SimurgArray component Primary reference Oracle Direct dependency OK?
IDAT v3 parser gtc2vcf idat reader (MIT); BeadArrayFiles (BSD-2) illuminaio (run in R), gtc2vcf --idat CSV dump Yes (MIT/BSD)
BPM / EGT / GTC readers BeadArrayFiles, Picard, gtc2vcf BeadArrayFiles Yes
GTC v5 writer gtc2vcf gtc_write (:1001-1152) + GTC_File_Format_v5.pdf Round-trip via BeadArrayFiles; md5 vs ACLI / the reference product Yes
Normalization 1.2.0 Patent US7035740 (expired), Illumina 2006 normalization note, GenTrain 3.0 tech note; gtc2vcf for bug-compatibility Compare the 13-float XForm per bin with the reference product's GTC NORMALIZATION_TRANSFORMS (TOC 400) Algorithm yes; copying code is OK under MIT with notice
GenCall scoring Kermani patents, Shen 2005; gtc2vcf Reference-product GTC GENOTYPE_SCORES (1004) same
GenTrain clustering (EGT creation) No OSS exists. Use the GenTrain 2.0/3.0 tech notes and US7467117 GenomeStudio (the reference product does not cluster) Must be built
BAF/LRR/LogRDev Peiffer 2006; gtc2vcf GTC 1012/1013/1008 Yes
Gender gtc2vcf v2 logic GTC 1007 Yes
GTC→VCF GTCtoVCF (Apache), Picard GtcToVcf (MIT), gtc2vcf Reference-product SNV VCF Yes
CNV PennCNV (public domain), mocha (MIT) PennCNV run; reference-product CNV VCF Yes
Contamination BAFregress (MIT) verifyIDintensity (GPL, subprocess) BAFregress yes
PGx star alleles PharmCAT (MPL-2.0), PyPGx (MIT) reference-product PGx output PharmCAT as unmodified dep; PyPGx yes

5.1 License-compatibility summary

  • Permissive, safe to vendor with notices: MIT (gtc2vcf, mocha, Picard, PyPGx, somalier, peddy, methylprep, sesame, illumina_array_reader), BSD-2 (BeadArrayFiles), BSD-3 (WARP), Apache-2.0 (GTCtoVCF; keep NOTICE), Unlicense (PennCNV).
  • Weak copyleft: MPL-2.0 (PharmCAT). Modified PharmCAT files must stay open; linking or bundling otherwise is OK. Artistic-2.0 (crlmm, minfi).
  • Strong copyleft (subprocess/oracle only): GPL-2 illuminaio; GPL-3 plink2, verifyIDintensity, GenotypeHarmonizer, akt, ASCAT; GADA (GPL).
  • Non-commercial or proprietary (avoid): argyle (CC BY-NC-SA), Aldy (IU), Cyrius (PolyForm Strict), Stargazer, QuantiSNP, iPattern, GenoSNP, optiCall (Sanger terms), zCall (no licence), EnsembleCNV (no licence visible).

6. Clean-room caveat and suggested process

Superseded (plan §2): SimurgArray may reuse MIT/BSD/Apache code with attribution, so no strict clean room is required, and black-box runs of the vendor binary are not part of the plan. The oracles are published vendor-produced GTCs (1000G Broad Omni2.5 set), gtc2vcf and BeadArrayFiles. Kept for context.

Illumina's normalization and GenCall algorithms are described in the expired patents (US7035740, US7467117) and in public tech notes. The bit-exact details (percentile interpolation, the robustfit adjfactor bug, the binned closest-point search, the ≥ 2 beads rule, norm-id byte wrap, FLT_MIN·FLT_EPSILON sentinels) are known mainly through gtc2vcf. gtc2vcf in turn states that it ports Illumina's GenCall C# source and AutoConvert behaviour.

MIT permits reuse, so there is no copyright barrier to reading gtc2vcf. If SimurgArray nonetheless wants a documented clean room (for example, to avoid any claim that it derives from Illumina's GenCall source), do this:

  1. Have a "dirty" analyst write a behavioural spec from patents, tech notes and black-box experiments on the reference product / IAAP output (GTC diffs).
  2. Have implementers code from that spec only.
  3. Use gtc2vcf only as a black-box test oracle: compare GTC TOC fields 400/1002/1004/1007/1008/1012/1013 per sample. Do not use it as source material for implementers.

Either way, validation must include a direct comparison against vendor-produced GTC output (now: the published 1000G Broad GTCs). gtc2vcf's equivalence claims stop at ACLI v2.1, and the reference product is not referenced in its code.

7. Sources

Reference in new issue View Git Blame Copy Permalink