kemal/SimurgArray
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
587fadbaefd885f6785caa1985cbe26bae889cbb
SimurgArray/docs/research/R1_reference_product.md
T
kemal 5695119e9c Add sanitized Phase-0 research archive under docs/research 
Research notes R1-R4, documentation-study files A-E (source of the
functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use
neutral naming and current project decisions (plan §2).
2026-09-27 16:27:34 +01:00

44 KiB
Raw Blame History

R1 — the reference product: public-source product dossier

Archived Phase-0 study (sanitized copy, see README.md). Written 2026-09-27, before the decisions in SIMURGARRAY_MASTER_PLAN.md §2. The analysed vendor product is called "the reference product" and its CLI refcli; its documentation URLs are replaced by [ref-docs:…] tokens. Where this file speaks of a clean room, legal clearance, or running/benchmarking the vendor binary, plan §2 supersedes it: personal research; MIT/BSD/Apache code may be reused with attribution; GPL tools are subprocess oracles only; vendor binaries are never run for comparison, inspected or decompiled; oracles are published vendor-produced GTCs, gtc2vcf and BeadArrayFiles.

Compiled 2026-09-27 for a reimplementation feasibility study. Only public sources were used. No binaries were downloaded, run, or inspected.

Tag legend:

  • [PUBLIC DOCUMENTATION]: stated in an Illumina (or other named public) source. The URL is given.
  • [INFERRED]: a direct logical consequence of public statements. The reasoning is shown.
  • [HYPOTHESIS]: a plausible guess that still needs validation.

Primary source: the official reference-product documentation site

  • Site: [ref-docs:home]. It is also mirrored under [ref-docs:mirror].
  • Every page also has a Markdown copy: append .md to the page URL. The site index is [ref-docs:llms.txt].
  • Versioned docs live under v1.0 to v1.3. The root path is v1.4, the latest release.

Short aliases used in this report:

Alias URL
[DOC-HOME] [ref-docs:readme]
[DOC-APPS] [ref-docs:our-features]
[DOC-LOCAL] [ref-docs:local-analysis]
[DOC-IN] [ref-docs:input-files]
[DOC-OUT] [ref-docs:output-files]
[DOC-QC] [ref-docs:local-analysis/qc-report]
[DOC-FAQ] [ref-docs:faq]
[RN-1.0] [ref-docs:release-notes-1.0.0]
[RN-1.1] [ref-docs:release-notes-1.1.0]
[RN-1.2] [ref-docs:release-notes-1.2.0]
[RN-1.3] [ref-docs:release-notes-1.3.0]
[RN-1.4] [ref-docs:release-notes-1.4.0]
[DOC-REV] [ref-docs:document-revision-history]
[DOC-V10-LOCAL] [ref-docs:v1.0/local-analysis]
[DL] [ref-docs:downloads]

1. What the reference product is

1.1 Identity

  • Name and scope. "[product name] secondary analysis is a … bioinformatics software for Illumina Infinium array-based assays." [PUBLIC DOCUMENTATION] [DOC-HOME]
  • Two packages.
    • A local package with a command-line interface: "no specialized server or hardware required".
    • A cloud package with a GUI, running in the vendor cloud platform.
    • [PUBLIC DOCUMENTATION] [DOC-HOME], [DOC-FAQ]
  • Binary name.
    • The CLI executable is refcli (refcli.exe on Windows). It sits in the refcli subfolder of the unpacked installer.
    • Version check: refcli version.
    • Example install path in the docs: /usr/local/bin/<refcli>-linux-x64-DAv1.4.0/<refcli>/.
    • [PUBLIC DOCUMENTATION] [DOC-LOCAL], Illumina Knowledge 000010127 ([ref-docs:kb-000010127])
  • Installer file names (from the download page):
    • …<Product>v1.4LinuxInstaller-1.4-<refcli>-linux-x64-DAv1.4.0-rc2-sha.9b051a4b….tar.gz (about 60 MB)
    • …-<refcli>-win-x64-DAv1.4.0-rc2-sha.9b051a4b….zip (about 57 MB)
    • Both dated May 4, 2026. Downloading requires a MyIllumina login and a customer or instrument serial number.
    • [PUBLIC DOCUMENTATION] [DL], Knowledge 000010127

1.2 Implementation technology

  • .NET application.
    • "[product] v1.4 is a self-contained .NET 9 application … Linux users may need to install native OS-level dependencies." [PUBLIC DOCUMENTATION] [RN-1.4], [DOC-LOCAL] (Linux Dependencies section)
    • A config file refcli.runtimeconfig.json exists. The docs suggest setting System.Globalization.Invariant=true in it as a workaround. [PUBLIC DOCUMENTATION] [RN-1.0]…[RN-1.4]
  • Internal namespaces. Stack traces published in every release note show:
    • ArrayAnalysis.CLI.App
    • ArrayAnalysis.Core.GtcFileLoader (source path /src/ArrayAnalysis.Core/GtcFileLoader.cs)
    • ArrayAnalysis.Services.GtcFactory
    • ArrayAnalysis.Services.GtcToVcfService
    • [PUBLIC DOCUMENTATION] [RN-1.0]
    • Inference: the refcli code base descends directly from the vendor's earlier array CLI (ACLI) code base. [INFERRED]

1.3 Release history

Version Date Key content Source
v1.0.0 Dec 2023 Local CLI: genotyping (IDAT→GTC→VCF), PGx CNV, native PGx star-allele caller with quality scores and CPIC/DPWG annotation. PGx validated only on GDA-ePGx (GDA_PGx…E2). [RN-1.0]
Genotyping Cloud 1.0.0 Mar 2024 Cloud genotyping; Final Report and Locus Summary outputs. [ref-docs:release-notes-1.0.0-cloud-genotype]
Methylation QC Cloud 1.0.0 May 2024 Cloud-only methylation QC: 21 control metrics, detection p-values, beta values, M-values. [ref-docs:release-notes-1.0.0-cloud-methylqc]
v1.1.0 Sep 2024 EX PGx chips (GSAv4-ePGx, GCRA-ePGx). More PGx genes. A custom EGT can be used with the stock CN model (warning only). JSON schema renames. [RN-1.1]
v1.2.0 Feb 2025 Whole-genome cytogenetics CNV/LOH calling, VCF output, BAF/LRR bedgraph, ISCN 2020 / cytoband annotation JSON. New pgx root command. New license server [ref-license-server]. [RN-1.2]
v1.3.0 Aug 2025 Mosaic-fraction estimation. PAR/sex-chromosome improvements. GenomeStudio-compatible sample sheet. Failed samples stay in gt_sample_summary. --smoothing removed from gtc-to-bedgraph. [RN-1.3]
Methylation QC Cloud 1.0.1 Feb 2026 Maintenance release. [ref-docs:release-notes-1.0.1-cloud-methylqc]
v1.4.0 May 2026 (latest) QC Report (qc call, qc report; "replacing the GenomeStudio control dashboard"). GAINLOH calling. Better mosaic detection. PGx SV star alleles and a new CSV set. --use-infI-nc-info for multi-allelic variant (MAV) calling. .NET 9. [RN-1.4]

All rows above are [PUBLIC DOCUMENTATION]. Additional notes:

  • The v1.0.0 cloud pipeline versions were deprecated in Q1 2026. [PUBLIC DOCUMENTATION] [RN-1.0]
  • Documentation revisions run from 01 (Dec 2023) to 14 (Aug 2026). Revision 14 renames the vendor cloud platform's data layer. [PUBLIC DOCUMENTATION] [DOC-REV]

1.4 Relationship to other Illumina tools

The vendor's earlier array CLI (ACLI)

IAAP (Illumina Array Analysis Platform) CLI and AutoConvert

  • These are earlier .NET GenCall front-ends (IDAT→GTC). [PUBLIC DOCUMENTATION] https://emea.support.illumina.com/downloads/iaap-genotyping-cli.html
  • Third-party gtc2vcf (MIT license) reports that "AutoConvert 2.0, IAAP-CLI, and ACLI … perform version 1.2.0 of the normalization step and seem to produce the exact same results", while the original AutoConvert uses normalization 1.1.2. [PUBLIC DOCUMENTATION – third party] https://github.com/freeseek/gtc2vcf
  • The reference product probably kept normalization 1.2.0 (or a successor) and GenTrain/GenCall-compatible calling. It still reads EGT cluster files and writes GTC v5, and GTCs from AutoConvert/Beeline are accepted by gtc-to-vcf as "legacy GTCs". [HYPOTHESIS], supported by [DOC-OUT]

GenomeStudio 2.0

  • It is still the tool for creating custom cluster files (EGT). The reference product does not cluster; it only consumes EGTs. [PUBLIC DOCUMENTATION] [DOC-LOCAL], section "Optimizing cluster files"
  • The reference-product QC Report "replaces GenomeStudio control dashboard". [PUBLIC DOCUMENTATION] [DOC-FAQ], [RN-1.4]
  • The cloud Final Report and Locus Summary are "similar to the one available in GenomeStudio". [PUBLIC DOCUMENTATION] [DOC-OUT]
  • For PGx cluster review, "GenomeStudio auxiliary files" (*_GS_import*.zip) are provided. [PUBLIC DOCUMENTATION] [DOC-LOCAL], [DL]

GTCtoVCF (github.com/Illumina/GTCtoVCF, Apache-2.0, Python 2.7)

  • Its README says: "As of May 2023, it is now recommended to use … [product] for GTC to VCF conversion." [PUBLIC DOCUMENTATION] https://github.com/Illumina/GTCtoVCF
  • refcli genotype gtc-to-vcf reuses GTCtoVCF's option names and near-verbatim documentation text:
    • --unsquash-duplicates, --auxiliary-loci, --filter-loci, --disable-genome-cache
    • the "Squashing duplicates", "Genome cache", "Auxiliary loci" and "combined genotype" paragraphs, and the dbSNP mapping conventions
    • Inference: the refcli implementation is a port or evolution of GTCtoVCF logic. That makes GTCtoVCF (open source, Apache-2.0) a strong legal reference implementation for the VCF conversion layer. [INFERRED] Compare [DOC-LOCAL] with the GTCtoVCF README.

The vendor cloud platform

  • The cloud edition of the reference product runs in the vendor cloud platform UI. Data is stored on the vendor cloud platform.
  • An iScan can stream IDATs to the cloud directly.
  • Analysis types in the vendor cloud platform dropdown:
    • "[product] – Genotyping and QC"
    • "– PGx – CNV calling"
    • "– PGx – Star allele annotation"
    • "– Cytogenetics analysis"
    • "- Cytogenetics analysis + tertiary interpretation" (cloud only)
    • "– Methylation – QC" (cloud only)
  • Tertiary-interpretation platform integration uses vendor cloud-platform SNS notifications.
  • [PUBLIC DOCUMENTATION] [DOC-APPS], [ref-docs:cloud-analysis], [ref-docs:cloud-analysis/cytogenetics-analysis]
  • Cloud sample limits: max 1152 samples for genotyping, cytogenetics and methylation; max 384 for the PGx types. [PUBLIC DOCUMENTATION] [DOC-APPS]

Relationship to the vendor's NGS secondary-analysis product of the same brand

  • The reference product shares only branding with it: no FPGA, a different code base, CPU only.
  • The developer article states: "The software runs on CPUs and does not require a specialized server or FPGA hardware."
  • [PUBLIC DOCUMENTATION] [ref-docs:dev-article-1.0-pgx]

2. Supported arrays and species

All of this section is [PUBLIC DOCUMENTATION] from [DOC-APPS] ("Product & Analysis Compatibility") unless noted.

2.1 Genotyping

"Any Infinium genotyping array, including human (including custom and semi-custom) and non-human diploid species".

Tested manifests:

Array Manifest Genome build
ASA ASA-24v1-0_E2 GRCh37/38
BovineSNP50 v3 BovineSNP50_v3_A1 UMD3 (non-human)
CGA CGA-24v1-0_A1 GRCh37/38
GCRA GCRA-24v1-0_20068388_B2 GRCh37/38
GDA GDA-8v1-0_D2 GRCh37/38
GSA v3 GSA-24v3-0_A2 GRCh37/38
GSA v4 (Infinium EX) GSA-48v4-0_20085471_D2 GRCh37/38
JSA JSA-24v1-0_B2 GRCh37/38
PRSbooster PRSbooster_20083382_A GRCh37

Species support:

  • Non-human diploid species work if the user supplies a reference FASTA. For non-human data, turning off VCF output is recommended on cloud.
  • "Genotyping only supports diploid organisms. Polyploid genotyping is currently not supported." [RN-1.1]…[RN-1.4]

2.2 PGx (CNV and star alleles)

Supported only for these commercial ePGx products, all on GRCh38:

  • GDA-ePGx GDA_PGx-8v1-0_20042614_G4
  • GSAv4-ePGx GSA-PGx-48v4-0_20079540_E2
  • GCRA-ePGx GCRA-PGx-24v1-0_20084467_C2

Semi-custom arrays get PGx calls only for the commercial backbone content. [DOC-LOCAL] (semi-custom section)

2.3 Cytogenetics (CNV/LOH)

Validated arrays:

  • GDA Cyto-8 (GDACyto-8v1-0_20047166_E2)
  • GSA Cyto-24 (GSACyto-24v1_20044998_C2)
  • CytoSNP-850K v1.4, iScan (_iScan_B2) and NextSeq550 (_NS550_B2, from v1.3)

Details:

  • Builds GRCh37 and GRCh38.
  • Constitutional samples only; "oncology samples not supported". [RN-1.2], [RN-1.4]
  • The feature list says "any human genotyping array" [RN-1.2], but custom products need a cyto model file from Tech Support. [DOC-IN]

2.4 Methylation

  • In scope, but only as cloud "Methylation QC". It is not available in the local CLI as a methylation pipeline. [DOC-APPS]
  • Supported arrays: EPIC v1 (B5 manifest), EPIC v2 (EPIC-8v2-0_A2), EPIC-Xtra (custom), and MSA (MSA-48v1-0_20102838_A1), all GRCh38.
  • The local refcli qc call --array-type methylation does compute methylation control-probe QC metrics, and qc report can report them. There is no local beta-value, M-value or detection-p pipeline. [PUBLIC DOCUMENTATION] [DOC-LOCAL], [DOC-QC]
  • Cloud methylation QC outputs, per sample:
    • {Barcode}_{Pos}_ctrl.tsv.gz (control probes)
    • {Barcode}_{Pos}_cgs.tsv.gz (beta values, M-values, detection p-values)
  • Cloud methylation QC outputs, per batch:
    • sample_qc_summary.{xlsx,tsv}
    • plots, including a PCA of beta values
    • [DOC-OUT]

2.5 Instruments

  • IDATs come from iScan. NextSeq550 IDATs are accepted for select arrays. [PUBLIC DOCUMENTATION] [DOC-IN]

3. Workflows and algorithms (what is and isn't disclosed)

3.1 Genotyping (IDAT → GTC → VCF)

What the docs state [PUBLIC DOCUMENTATION]:

  • Pipeline: genotype call (IDAT + BPM + EGT → GTC v5), then genotype gtc-to-vcf (GTC + BPM + CSV manifest + FASTA → per-sample *.snv.vcf.gz + .tbi).
  • --gencall-cutoff defaults to 0.15 ("GenCall score cutoff to label a NoCall"). [DOC-LOCAL]
  • The GTC holds genotypes, GenCall scores, BAF, LRR and sample QC. It is the "Binary proprietary format", parseable with https://github.com/Illumina/BeadArrayFiles. [DOC-OUT]
  • In the SNV VCF, FILTER is always PASS. FORMAT fields are GT:GS:BAF:LRR. Genotypes are ploidy-adjusted: haploid on Y, MT and non-PAR X in males. [DOC-OUT]
  • BAF and LRR are oriented REF=A / ALT=B relative to the reference. They may therefore be 1 − the GenomeStudio value. [DOC-OUT]
  • Multi-allelic / duplicate-site merging:
    • Enumerate the candidate genotypes from the queried alleles.
    • Filter out genotypes inconsistent with each assay's call, respecting design limits. For example, Infinium II cannot tell A/T or C/G apart.
    • Exactly one survivor becomes the call; zero or more than one gives a no-call.
    • For merged records, GS is the minimum GS across assays. MAV records carry no BAF or LRR.
    • --use-infI-nc-info (v1.4) treats Infinium I no-calls as "missing signal" rather than "probe failure", which removes alleles. It requires a cluster file with "zeroed-out" MAV assays.
    • [DOC-OUT], section "Note on Multi-Allelic Variants"
  • Probe IDs in the ID field are joined with "," for SNPs and ";" for indels. [DOC-OUT]

Algorithms not disclosed for the reference product itself: GenCall normalization and calling.

  • The GTC stores per-sample NormalizationTransform records: version, offset_x/y, scale_x/y, shear, theta. The public BeadArrayFiles code shows how they are applied: subtract offset → rotate by theta → shear → scale → clamp ≥0. It then computes R = x+y and θ = atan2(y,x)·2/π. [PUBLIC DOCUMENTATION] https://github.com/Illumina/BeadArrayFiles/blob/develop/module/GenotypeCalls.py
  • How the transform parameters are fit is only described qualitatively in the tech notes (see §8). [PUBLIC DOCUMENTATION]

3.2 PGx copy number (targeted)

Scope [PUBLIC DOCUMENTATION] [ref-docs:pharmacogenomics-pgx-reference/pgx-cnv-coverage]

  • 7 genes across 10 regions (GRCh38): GSTM1, UGT2B17, CYP2E1, SULT1A1, CYP2A6 (intron.7, exon.1), CYP2D6 (exon.9, intron.2, p5), and GSTT1 (on 22_KI270879v1_alt).

Batch-based calling [PUBLIC DOCUMENTATION] [DOC-LOCAL]

  • Needs ≥24 samples, with ≥22 passing LogRDev < 0.2. ≥96 are recommended; about 500 GDA-ePGx samples fit per batch on the reference hardware.
  • Error text when too few pass: "{n} sample(s) passed QC. Requires at least {m} samples". [DOC-OUT]

Model file and training [PUBLIC DOCUMENTATION] [DOC-LOCAL], [DOC-IN]

  • Requires a manifest-revision-specific CN model .dat.
  • The model is trainable with pgx copy-number train:
    • needs ≥96 samples, of which ≥90 pass LogRDev ≤ 0.2; ≥150 are recommended, with diminishing returns beyond 3,000
    • needs --platform LCG|EX
    • needs a .msk mask file next to the BPM, plus the FASTA
  • The trainer "is designed with the assumption that the copy number distribution resembles the standard population distributions".
  • v1.4 turns an EGT mismatch between GTC and CN model into a hard error. v1.1–1.3 only warned. [RN-1.4]

Output [PUBLIC DOCUMENTATION] [DOC-OUT]

  • *.cnv.vcf.gz with <CNV> records, FORMAT CN:NR. CN is capped at 5 ("5 or 5+"). NR is the "aggregated normalized intensity".
  • QUAL is Phred-scaled, capped at 60. Q7 filter below 7; SampleQuality filter when LogRDev > 0.2.
  • Header keys ##OverallPloidy= and ##GCCorrect=True, which implies GC correction of intensities.
  • Batch files: cn_batch_summary.csv (gain/loss/neutral per region) and a CN summary file.

Algorithm [HYPOTHESIS]

  • Not disclosed. The developer article gives accuracy only: CYP2D6 CNV "98.3%–99.0%" versus qPCR on 288 samples.
  • Likely shape: batch-normalized, GC-corrected, probe-aggregated normalized R (the "NR" field), compared against per-region reference distributions in the .dat model (population-trained). Plausibly a Gaussian-mixture or likelihood model over CN states 0–5.

3.3 PGx star alleles

Commands and licensing [PUBLIC DOCUMENTATION] [DOC-LOCAL]

  • pgx star-allele call (VCF folder with *.snv.vcf.gz + *.cnv.vcf.gz, plus a PGx database .zip) → star_alleles.dat.
  • Then pgx star-allele annotate --guidelines CPIC|DPWG → CSV and JSON.
  • Star-allele calling is the only paid, license-server-gated local step. It is charged per sample through --license-server-url https://user:pass@[ref-license-server] (v1.2+), or [ref-license-server-legacy] for v1.0/1.1.

Data sources [PUBLIC DOCUMENTATION] [ref-docs:pharmacogenomics-pgx-reference/pgx-allele-definitions-and-pgx-guidelines]

  • PharmVar 6.1, PharmGKB snapshot 2024-05-16, UGT nomenclature, legacy CYP nomenclature, CPIC 1.38.0, DPWG June 2023.
  • CPIC phenotypes cover 17 genes. Activity scores are given for CYP2C9, CYP2D6 and DPYD.
  • Coverage: "2400+ targets for over 50 genes" [DOC-HOME]; the product page says "1700+ targets for 50+ genes".

Scoring and reporting [PUBLIC DOCUMENTATION] [RN-1.0], [DOC-OUT]

  • Quality score is a "negative log likelihood" in v1.0.
  • Later, "a probability score … based on the SNV and CNV inputs, input quality, and population frequencies" (0–1), plus a Raw Score that excludes population frequency.
  • Ranked alternative solutions; hybrid alleles; allele-specific CN.
  • Alternatives are shown if their score is ≥50% of the top score.
  • No novel alleles are called.

PGx database format [PUBLIC DOCUMENTATION] [DOC-IN]

  • "each line … a single probe ID mapping to a variant's HGVS tag". The public database zip is named DAv1.4.0-rc2-pgx-mapping-sha.b164b4aa….zip. [DL]

v1.4 CSV outputs [PUBLIC DOCUMENTATION] [DOC-OUT]

  • diplotype_summary.csv
  • diplotype_annotations.csv
  • supporting_variants.csv
  • unresolved_variants.csv
  • variants_not_covered_on_array.csv
  • plus a per-sample star-allele JSON. Its fields are listed in [DOC-OUT].

Algorithm [HYPOTHESIS]

  • Not disclosed. It is described only as "a new and native PGx star allele caller" (developer article).
  • Possible approach: a haplotype-pair likelihood search over PharmVar definitions. It would take CN constraints from the CNV VCF and error or missing-data tolerance, and use population-frequency priors, which would explain the "Score" versus "Raw Score" split.
  • The v1.x error "The model has been changed since the solution was last computed" is typical of Google OR-Tools / a MIP/CP solver. That suggests an optimization (ILP/CP-SAT) formulation. [HYPOTHESIS]

3.4 Cytogenetics CNV/LOH (genome-wide)

Commands [PUBLIC DOCUMENTATION] [DOC-LOCAL]

  • cyto call --cn-model cyto_model.dat (GTC → *.cnv.vcf.gz + cyto.cnv.dat).
  • cyto annotate --annotation-db CytoAnnotateData_*.zip (→ per-sample JSON with ISCN 2020 notation, cytobands, RefSeq/Ensembl genes, QC).

Model file [PUBLIC DOCUMENTATION] [DOC-IN]

  • The cyto model "primarily contains probe GC content information and is tied to the probe sequence/manifest".
  • It does not need retraining when the EGT changes.

Parameters [PUBLIC DOCUMENTATION] [DOC-LOCAL]

Event type Default minimum probes Default minimum size
DEL 10 25 kb
DUP 10 50 kb
LOH 500 3000 kb
GAINLOH 300 500 kb
mosaic DEL (mdel) 10 25 kb
mosaic DUP (mdup) 10 50 kb
mosaic LOH (mloh) 500 3000 kb

Other parameters:

  • --smoothing 5 (probes on each side)
  • --max-mosaic-fraction 1.0
  • --min-qual 20 (annotate)
  • Minimum probe thresholds below 10 are reset to 10.

Behaviours documented [PUBLIC DOCUMENTATION] [DOC-LOCAL], [DOC-OUT], [RN-1.2]…[RN-1.4]

  • CN up to 4+.
  • QUAL = −10·log10(1 − P(called state)), capped at 60; Q10 filter.
  • LogRDev > 0.2 → SampleQuality flag. LogRDev > 0.3 → only whole-chromosome events are reported.
  • Mosaic events are detected down to about 15% for one-copy mosaic states. MF holds the mosaic fraction.
  • LOHTYPE ∈ {AOH, CNLOH, GAINLOH}.
  • LOH < 500 kb flanked by copy-neutral regions is removed (v1.4).
  • Variants must match the expected LRR sign (v1.4).
  • Effective size excludes gaps larger than 150× the median probe spacing.
  • An unknown sex is resolved by the cyto caller or else assumed male.
  • GT is hardcoded to 1/1.

Algorithm [HYPOTHESIS]

  • Not disclosed. The outputs (per-state posterior → Phred QUAL; joint LRR+BAF states including LOH, GAINLOH and mosaic fractions; GC correction; smoothing window) are consistent with an HMM or segmentation + state-likelihood model over LRR and BAF, in the family of PennCNV / cnvPartition / the reference product "cytogenetics modality".
  • cnvPartition is Illumina's older published recursive-partition approach (see §8).

3.5 QC (v1.4)

[PUBLIC DOCUMENTATION] [DOC-LOCAL], [DOC-QC], [DOC-OUT]

  • qc call --array-type genotyping|methylation → controls.raw_metrics.csv, controls.qc_metrics.csv. These files are also emitted automatically by genotype call.
  • qc report → <Product>_QC_Report_YYYY_MM_DD.html (self-contained, interactive) and <Product>_QC_table_YYYY_MM_DD.{xlsx|csv}.

3.6 Other items

  • Locus Report / Final Report / Locus Summary.
    • Cloud-only outputs: gtc_final_report.csv (GenomeStudio-style) and locus_summary.csv, with columns Call_Freq, Het_Excess_Freq, ChiTest_P100, Cluster_Sep, AA/AB/BB θ and R mean/SD, and so on.
    • No locus report exists in the local CLI.
    • [PUBLIC DOCUMENTATION] [DOC-OUT], [DOC-APPS]
  • Cluster file generation. Not supported; it is done in GenomeStudio 2.0. [PUBLIC DOCUMENTATION] [DOC-LOCAL]
  • BedGraph. genotype gtc-to-bedgraph → *.LRR.bedgraph, *.BAF.bedgraph for IGV. [PUBLIC DOCUMENTATION] [DOC-LOCAL], [RN-1.2]

4. Command-line interface (v1.4, exact)

Syntax: refcli [module] [sub-module] [command] [required] [optional]. Options are case-sensitive. [PUBLIC DOCUMENTATION] [DOC-LOCAL]

The following commands are all [PUBLIC DOCUMENTATION] from [DOC-LOCAL]:

refcli version | help
refcli qc call        --array-type {genotyping|methylation} --csv-manifest F (--idat-folder D | --sample-sheet F) [--output-folder] [--debug] [--json-log]
refcli qc report      --data D1[,D2..|PARENT] [--config cfg.yaml] [--label L1,..] [--output-folder] [--output-format xlsx|csv]
                       [--threshold-callRate 0.98] [--threshold-logRdev 0.20] [--debug] [--json-log]
refcli genotype call  --bpm-manifest F.bpm --cluster-file F.egt (--idat-folder D | --sample-sheet F) [--gencall-cutoff 0.15]
                       [--num-threads N] [--output-folder] [--debug] [--json-log]
refcli genotype gtc-to-bedgraph --bpm-manifest F.bpm (--gtc-folder D | --sample-sheet F) [--output-folder] ...
refcli genotype gtc-to-vcf --bpm-manifest F.bpm --csv-manifest F.csv --genome-fasta-file G.fa (--gtc-folder D | --sample-sheet F)
                       [--auxiliary-loci aux.vcf] [--disable-genome-cache] [--filter-loci list.txt] [--unsquash-duplicates]
                       [--no-bgzip] [--use-infI-nc-info] [--output-folder]
refcli pgx copy-number call  --cn-model M.dat (--gtc-folder D | --sample-sheet F) [--no-bgzip] [--output-folder]
refcli pgx copy-number train --bpm-manifest F.bpm (mask .msk beside it) --genome-fasta-file G.fa --platform {LCG|EX}
                       (--gtc-folder D | --sample-sheet F) [--disable-genome-cache] [--output-folder]
refcli pgx star-allele call  --database pgx.zip --license-server-url URL --vcf-folder D [--query-license-quota] [--output-folder]
refcli pgx star-allele annotate --star-alleles star_alleles.dat [--guidelines CPIC|DPWG] [--output-folder]
refcli cyto call      --cn-model cyto_model.dat (--gtc-folder D | --sample-sheet F) [--no-bgzip] [--smoothing 5]
                       [--min-{del,dup,loh,gainloh,mdel,mdup,mloh}-{probes,size} ...] [--max-mosaic-fraction 1.0] [--output-folder]
refcli cyto annotate  --annotation-db CytoAnnotateData.zip --vcf-folder D [--min-* ...] [--min-qual 20] [--output-folder]

Common flags on every command: --debug, --json-log, --help, --version.

Behaviour notes:

  • Output folders are overwritten on re-run.
  • A --idat-folder is searched recursively.
  • Duplicate IDAT pairs per barcode_position are unsupported.

4.1 Inputs

[PUBLIC DOCUMENTATION] [DOC-IN]

Input Extension Used by
IDAT (Grn/Red pair per Barcode_RxxCxx) .idat genotype call, qc call
BPM manifest .bpm genotype call/gtc-to-vcf/gtc-to-bedgraph, pgx cn train
CSV manifest (with SourceSeq, RefStrand; dbSNP-style 1-based mapping) .csv gtc-to-vcf, qc call
Cluster file .egt genotype call
PGx CN model .dat pgx copy-number call
Cyto model .dat cyto call
Mask .msk (beside the BPM) pgx copy-number train
PGx database .zip star-allele call
Cyto annotation database (RefSeq + Ensembl) .zip cyto annotate
Genome FASTA + .fai; contigs "1", not "chr1" .fa gtc-to-vcf, pgx cn train
QC config .yaml / .yml qc report

Sample sheet (CSV):

  • Keys are SentrixBarcode_A,SentrixPosition_A. On cloud they are beadChipName,sampleSectionName.
  • Optional [Header] with RootFolder and [Data] sections. The format is compatible with GenomeStudio sheets, and [Manifests] is ignored.
  • Column names are case-insensitive, and reserved names are rejected.
  • Values are limited to 500 characters.
  • Sample-name precedence: sheet Sample_Name > IDAT/GTC metadata > Barcode_Position.

Public product files: GRCh37/GRCh38 FASTA zips, per-product BPM/CSV/EGT, CN/cyto model and database zips, QC YAML templates, and demo IDATs.

  • Examples: GDA-ePGx demo data set (iScan), 15.4 GB; GCRA/GSA-PGx idats_Demo_36.zip.
  • [PUBLIC DOCUMENTATION] [DL]
  • Note: model and database files are Illumina data files; their licence terms were not reviewed.

4.2 Outputs

[PUBLIC DOCUMENTATION] [DOC-OUT]

Per sample

  • .gtc (GTC v5)
  • *.snv.vcf.gz + .tbi. Header ##source=refcli 1.4.0, VCFv4.1. Tabix output is not byte-identical to bcftools index.
  • *.LRR.bedgraph, *.BAF.bedgraph
  • PGx *.cnv.vcf.gz
  • cyto *.cnv.vcf.gz; header ##source=refcli 1.4.0 Cyto, ##product=…
  • cyto annotation *.json
  • star-allele JSON
  • methylation (cloud only): _ctrl.tsv.gz, _cgs.tsv.gz

Per batch

  • gt_sample_summary.csv and .json. Columns: Sample ID, Sample Name, Sample Folder, Autosomal Call Rate, Call Rate, Log R Ratio Std Dev, Sex Estimate, [TGA_Ctrl_5716 Norm R], <samplesheet columns>.
  • controls.raw_metrics.csv, controls.qc_metrics.csv
  • <Product>_QC_Report_*.html, <Product>_QC_table_*.xlsx|csv
  • cn_batch_summary.csv, CN summary
  • star_alleles.dat
  • the five PGx CSVs (§3.3)
  • cyto.cnv.dat
  • cloud only: gtc_final_report.csv, locus_summary.csv, sample_qc_summary.{xlsx,tsv}

Logs

  • Warnings and errors go to the terminal, optionally as JSON (--json-log).
  • Error strings are catalogued in [DOC-OUT], for example "Skipping intensity only locus", "Failed to combine genotypes due to inconsistencies …", and "Cluster file (…) is not the same as CN Model Cluster file".

5. QC metrics and thresholds

5.1 Functional per-sample metrics (gt_sample_summary)

[PUBLIC DOCUMENTATION] [DOC-OUT], [DOC-QC], [DOC-FAQ]

  • Metrics: Autosomal Call Rate, Call Rate, Log R Ratio Std Dev (LogRDev), Sex Estimate (M/F/Unknown), and TGA_Ctrl_5716 Norm R.
  • TGA_Ctrl_5716 Norm R is present on PGx products only. It is the normalized R of one probe; a value < 1 means the targeted gene amplification (TGA) failed.
  • Failed samples get call rates of 0 and NaN for LogRDev (v1.3+).

Documented thresholds [PUBLIC DOCUMENTATION]

Metric / rule Value Source
qc report default callRate 0.98 (PASS if Autosomal Call Rate ≥ threshold) [DOC-LOCAL], [DOC-QC]
qc report default logRDev 0.20 (PASS if ≤) [DOC-LOCAL], [DOC-QC]
Generic genotyping YAML example logRDev 0.15 [DOC-QC]
PGx CNV: per-sample pass LogRDev < 0.2 (≤ 0.2 for training) [DOC-LOCAL]
PGx CNV: batch minimum ≥ 22 of ≥ 24 passing [DOC-LOCAL]
PGx star-allele: review triggers LogRDev > 0.2, call rate < 0.99, TGA control < 1.0 [DOC-LOCAL]
Cyto: low-quality flag LogRDev > 0.2 → SampleQuality [DOC-LOCAL]
Cyto: whole-chromosome-only mode LogRDev > 0.3 [DOC-LOCAL]

Not defined in the reference-product docs

  • The exact Sex Estimate algorithm.
  • p10 GC / p50 GC ("GC10"/"GC50") and p05/p50/p95 percentiles.
    • These are GTC fields (see BeadArrayFiles TOC IDs 1009, 1011, 1014, 1015), but they are not reported in gt_sample_summary. [PUBLIC DOCUMENTATION] https://github.com/Illumina/BeadArrayFiles
    • "p99 GC" was not found in any reference-product document.
  • LogRDev computation details, such as which loci are included and whether outliers are trimmed. The cyto JSON defines it as "Standard deviation of the Log R ratio values for all probes". [PUBLIC DOCUMENTATION] [DOC-OUT]

5.2 Control-probe QC metrics (controls.qc_metrics.csv)

[PUBLIC DOCUMENTATION] [DOC-OUT]

Definitions:

  • bkg_grn = max green intensity of Extension(A|T).
  • bkg_red = max red intensity of Extension(C|G).
  • Offset x = 3000.

Genotyping metrics:

Metric Formula
RestorationQC Restoration green / (bkg_grn + x)
StainingGreenQC Biotin(High) / Biotin(Bgnd)
StainingRedQC DNP(High) / DNP(Bgnd)
ExtensionGreenQC min Ext(C,G) green / max Ext(A,T) green
ExtensionRedQC min Ext(A,T) red / max Ext(C,G) red
Hyb High/Med and Hyb Med/Low green ratios
TargetRemovalIQC (bkg_grn + x) / TargetRemoval green
StringencyQC String(PM) red / String(MM) red
NonSpecificBinding Green / Red (bkg + x) / max NSB(Bgnd)
NonPolymorphic Green / Red min NP(C,G) / max NP(A,T) green; min NP(A,T) / max NP(C,G) red

Methylation adds:

  • TargetRemovalII
  • Specificity I green/red, Specificity II, Specificity II background
  • Bisulfite Conversion I green/red (with background variants), Bisulfite Conversion II (with background variant)
  • that is, the 21 BeadArray Controls Reporter metrics.

Genotyping YAML template example values [DOC-QC]:

Threshold Example value
staining 5
extension 5
targetRemovalI 5
hyb ratios 1.25
NP 5
stringency 1.5
NSB 5
tgaControl 1
restoration null (1 when the FFPE Restore kit is used)
  • Chemistry-specific templates: config_genotyping.yaml, config_genotyping_EX.yaml, config_methylation_EPIC.yaml, config_methylation_MSA.yaml (downloads on [DL]).
  • Methylation cloud defaults (EPIC / MSA) are tabulated at [ref-docs:cloud-analysis/methylation-qc]. Includes BgCorrectionOffset 3000 and PvalThreshold 0.05.
  • Methylation algorithm [PUBLIC DOCUMENTATION] from tech note M-GL-02644 ([ref-docs:methylation-qc-tech-note-M-GL-02644]):
    • Noob background correction
    • nonlinear dye-bias correction
    • detection p-values using "Negative controls for null, log transformed signal"
    • GenomeStudio, by contrast, uses background subtraction, linear dye bias, and a negative-control null.

6. Hardware, OS, performance

[PUBLIC DOCUMENTATION] [DOC-LOCAL], [DOC-FAQ], developer article

  • CPU only. "does not require a specialized server or FPGA hardware". The local package is "CPU-based".
  • Recommended machine.
    • 8 cores.
    • Memory: 32 GB in v1.4; 16 GB in v1.0 [DOC-V10-LOCAL].
    • ≥30 GB disk; cyto annotate needs ≥5 GB free.
  • Operating systems. Windows 10+ (win10-x64); CentOS 7+ / Ubuntu 20.04+ (linux-x64).
    • .NET 9 self-contained, needing native .NET dependencies.
    • Should not run as admin.
    • No ARM build is listed. [INFERRED] from the runtime IDs listed (win10-x64, linux-x64).
  • Docker.
  • Performance claims.
    • "Genotyping VCF in as little as 35 seconds per sample". [DOC-HOME]
    • "all PGx analysis results in ~1 minute per sample". [DOC-HOME]
    • Accuracy >99.5%. [DOC-HOME]
    • PGx CNV >95%. [DOC-HOME]
    • CYP2D6 call rate >97%. [DOC-HOME] The product page says >98%.
  • Threading and scale.
    • --num-threads exists on genotype call.
    • More than 1K samples per sample sheet slows things down (known issue).
    • Windows and Linux star-allele results differ slightly: 6 field differences in 1,576 samples, with concordant top solutions. [RN-1.3]
  • Network requirement.
    • Internet access is needed for the license check, only for pgx star-allele call.
    • Everything else runs offline. [INFERRED] Only star-allele call is quota-gated per [DOC-LOCAL] and [DOC-APPS] ("Local: No cost download" for genotyping, CNV and cyto).

7. Licensing / EULA

  • Product-specific EULA text was not found at a public URL. It is presumably shown in the installer or on MyIllumina download. [HYPOTHESIS] Action: obtain it legally and have counsel review it before any benchmarking with the real binary. (Superseded: the plan does not run, benchmark or inspect the vendor binary; see plan §2.)
  • Commercial model.
    • Local genotyping, PGx CNV and cyto are a "No cost download".
    • Local star-allele calling uses per-sample quota.
    • Cloud uses vendor cloud credits, plus a per-sample PGx charge and a vendor cloud-platform subscription.
    • [PUBLIC DOCUMENTATION] [DOC-APPS], product page [ref-docs:product-page]
  • Representative vendor EULA language [PUBLIC DOCUMENTATION] the vendor's sequencing-pipeline v4.3 EULA, [ref-docs:vendor-sequencing-eula-v4.3]. This is the same family and is likely similar for the reference product [HYPOTHESIS]:

    "Licensee may not modify, improve, translate, reverse engineer, decompile, disassemble, or create derivative works of the Software or otherwise attempt to … (b) derive the source code or the underlying ideas, algorithms, structure, or organization form of the Software."

    • The same EULA says Licensee may not "publish" the Software; the restriction is on the Software itself, not on results.
    • No explicit benchmarking or publication-of-results clause was found in the passages reviewed. [PUBLIC DOCUMENTATION] same PDF.
  • Implications [INFERRED]
    • A clean-room effort should rely only on public docs, public tech notes, open-source Illumina code, and observed input/output formats.
    • It must not decompile the .NET assemblies. .NET is easily decompilable, which makes this contractually sensitive.
    • Running the licensed binary to generate comparison outputs is probably permitted "internal use", but the "derive … underlying ideas, algorithms" clause is broad. That needs legal review. (Superseded: the plan does not run, benchmark or inspect the vendor binary; see plan §2.)
  • Open-source Illumina pieces usable as references (§9):
    • BeadArrayFiles: BSD-2-style licence text in its README.
    • GTCtoVCF: Apache-2.0.

8. Illumina public algorithm documents (URLs verified live, HTTP 200)

Document URL Relevance
Illumina GenCall Data Analysis Software (tech spotlight) https://www.illumina.com/Documents/products/technotes/technote_gencall_data_analysis_software.pdf Per-array self-normalization (channel intensity, background, dye crosstalk). GenTrain clustering with neural-net estimation of missing clusters. Bayesian calling. GenCall score = GenTrain score × data-to-model fit, penalised by the DNA score. GC10/GC50. Sensitive range 0.2–0.7.
Infinium Genotyping Data Analysis (tech note) https://www.illumina.com/Documents/products/technotes/technote_infinium_genotyping_data_analysis.pdf (alt: https://support.illumina.com/content/dam/illumina-marketing/documents/services/technote_infinium_genotyping_data_analysis.pdf) Normalization, clustering, QC workflow, and cluster-editing practice. Cited by the reference product docs [DOC-LOCAL].
GenTrain 2.0 https://www.illumina.com/Documents/products/technotes/technote_gentrain2.pdf Three-step θ/R clustering: preliminary, model proposals, HWE-aware scoring.
GenTrain 3.0 (M-GL-01258) https://www.illumina.com/content/dam/illumina/gcs/assembled-assets/marketing-literature/gentrain-tech-note-m-gl-01258/gentrain-tech-note-m-gl-01258.pdf Normalization "bins" (Inf I and Inf II separate). A two-parameter model for Inf I bins with < 192 loci, versus six parameters otherwise. Limits the penalty on near-axis homozygous clusters.
X-chromosome SNP cluster generation https://www.illumina.com/documents/products/technotes/technote_x_chromosome_snp_cluster_generation.pdf Sex-aware clustering.
DNA Copy Number Analysis Algorithms (cnvPartition basis) https://www.illumina.com/documents/products/technotes/technote_cnv_algorithms.pdf LRR/BAF definitions; recursive-partition CNV.
Cytogenetic analysis tech note https://www.illumina.com/Documents/products/technotes/technote_cytoanalysis.pdf LRR/BAF cytogenetics.
cnvPartition in GenomeStudio (training PDF) https://support.illumina.com/content/dam/illumina-support/courses/GenomeStudio-cnvPartition/story_content/external_files/GenomeStudio_cnvPartition.pdf
CNV plug-ins for BeadStudio https://www.illumina.com/content/dam/illumina-marketing/documents/products/technotes/technote_cnv_plug_ins.pdf
Custom cluster file creation for improved copy number analysis (M-GL-02142) https://www.illumina.com/content/dam/illumina/gcs/assembled-assets/marketing-literature/custom-cluster-file-tech-note-m-gl-02142/custom-cluster-file-tech-note-m-gl-02142.pdf Cited by the reference product for EGT/CN-model retraining.
[product] Methylation QC (M-GL-02644) [ref-docs:methylation-qc-tech-note-M-GL-02644] Noob, nonlinear dye bias, detection-p method.
iSelect / Infinium I vs II design https://www.illumina.com/Documents/products/technotes/technote_iselect_design.pdf Needed for MAV logic.
Infinium Controls training guide https://support.illumina.com/content/dam/illumina-support/courses/eval-inf-controls/story_content/external_files/Infinium_Controls_Training_Guide.pdf Control-probe semantics.
Developer article: [product] 1.0 PGx [ref-docs:dev-article-1.0-pgx] Accuracy numbers; CPU-only.
  • Tech-note summaries in the table are [PUBLIC DOCUMENTATION], extracted from the PDFs with pdftotext.
  • Key gap: none of these gives equations for the normalization-transform fitting, the GenCall data-to-model score, or GenTrain's neural-net cluster imputation.
  • Published third-party descriptions and reimplementations of Illumina-like calling exist: gtc2vcf (MIT), which re-implements BAF/LRR from EGT; and the WARP Illumina genotyping pipeline, https://broadinstitute.github.io/warp/docs/Pipelines/Illumina_Genotyping_Arrays_Pipeline/README (it uses iaap-cli). [PUBLIC DOCUMENTATION]

9. Illumina GitHub repositories relevant to arrays

The GitHub API was rate-limited (unauthenticated), so repos were checked through their public HTML/raw pages. A complete org listing still needs to be done (gh api orgs/Illumina/repos --paginate with a token).

Repo Licence Relevance
Illumina/BeadArrayFiles — https://github.com/Illumina/BeadArrayFiles BSD-style (text in README, "Copyright (c) 2020, Illumina") Python parsers for GTC (v5 spec: docs/GTC_File_Format_v5.pdf), BPM, EGT, and LocusAggregate. NormalizationTransform.normalize_intensities and rect_to_polar give the exact application of the stored affine normalization. The README recommends the reference product for making GTCs. [PUBLIC DOCUMENTATION]
Illumina/GTCtoVCF — https://github.com/Illumina/GTCtoVCF Apache-2.0 Semi-archived since May 2023 in favour of the reference product. Same option set and merge semantics as refcli genotype gtc-to-vcf (§1.4). Includes unit tests (tests/test_class.py) on genotype merging. [PUBLIC DOCUMENTATION]
Illumina/IlluminaHumanMethylationEPICv2manifest, Illumina/IlluminaHumanMethylationMSAmanifest not checked R/Bioconductor-style manifests for EPIC v2 and MSA; useful for methylation QC. [PUBLIC DOCUMENTATION] (the repos exist)
Illumina/Cyrius "Other" licence (not SPDX-identified) CYP2D6 caller for WGS, not arrays. Conceptual reference for CYP2D6 star-allele and CN logic only.
Illumina/canvas, Illumina/SMNCopyNumberCaller, Illumina/Gauchian, Illumina/ExpansionHunter, Illumina/hap.py, Illumina/witty.er "Other" licence (not SPDX-identified) NGS tools. hap.py and witty.er are useful for benchmarking VCF concordance.

Related non-Illumina open source:

10. Biggest unknowns and risks for an independent reimplementation

  1. GenCall internals (normalization fitting, per-locus calling likelihood, GenCall score formula, DNA-score penalty, sex estimation).
    • Only qualitative tech notes are public.
    • Reproducing bit-compatible GTC genotypes and GC scores will need empirical calibration against reference outputs.
    • The GTC stores the per-sample normalization transform, which helps with verification.
  2. PGx CNV model (.dat format and algorithm) and cyto model (.dat: "primarily … GC content"). Both formats and both algorithms are undocumented.
  3. Star-allele caller: scoring function, priors, solver, hybrid-allele handling. The PGx database zip format is only loosely described ("probe ID → HGVS" lines).
  4. Cyto CNV/LOH caller: the segmentation or HMM approach, the mosaic-fraction estimator, GC correction, and QUAL calibration are all undocumented.
  5. LogRDev, Autosomal Call Rate and Sex Estimate: the exact locus sets and formulas are not specified.
  6. EULA:
    • The product-specific terms are not public.
    • The vendor-family EULA forbids deriving "underlying ideas, algorithms" from the Software.
    • Comparison benchmarking should be legally cleared. (Superseded: the plan does not run, benchmark or inspect the vendor binary; see plan §2.)
    • .NET decompilation must be strictly avoided.
  7. Clearly reproducible from public material: VCF conversion (GTCtoVCF), GTC/BPM/EGT I/O (BeadArrayFiles), control-probe QC formulas (fully tabulated in [DOC-OUT]), methylation Noob/dye-bias/beta/M-value methods (open literature plus the tech note), and output schemas (VCF headers, CSV/JSON fields).
Reference in new issue View Git Blame Copy Permalink