Research notes R1-R4, documentation-study files A-E (source of the functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use neutral naming and current project decisions (plan §2). Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
59 KiB
A — Local CLI & Input Files: exhaustive analysis
Archived Phase-0 study (sanitized copy, see
README.md). Written 2026-09-27, before the decisions inSIMURGARRAY_MASTER_PLAN.md§2. The analysed vendor product is called "the reference product" and its CLIrefcli; its documentation URLs are replaced by[ref-docs:…]tokens. Where this file speaks of a clean room, legal clearance, or running/benchmarking the vendor binary, plan §2 supersedes it: personal research; MIT/BSD/Apache code may be reused with attribution; GPL tools are subprocess oracles only; vendor binaries are never run for comparison, inspected or decompiled; oracles are published vendor-produced GTCs, gtc2vcf and BeadArrayFiles.
Sources (both read in full, line by line):
site/pages/product-guides/local-analysis.md(647 lines, "[product] Local Analysis")site/pages/product-guides/input-files.md(205 lines, "Input Files")
Conventions in this document: "the vendor tool" = the analysed product; executable name is refcli.
Line references L:n = local-analysis page, I:n = input-files page. Verbatim quotes are in code or "quotes".
Items marked [DOC-INCONSISTENCY] are contradictions/omissions in the source docs themselves.
Images: neither page contains any image reference (no ![], <img>, .png/.jpg/.svg, or gitbook/assets URLs).
Only external hyperlinks exist (listed in §8).
1. Complete command reference
1.1 General syntax (L:143-147)
refcli [module] [sub-module (not needed for cyto)] [command] [required parameters] [optional parameters]
- Command names are lowercase; CLI options are case-sensitive and must be typed exactly, notably mixed-case
--threshold-callRateand--threshold-logRdev(L:147, repeated L:564). - Top-level modules (L:149-156):
| Module | Description (verbatim) |
|---|---|
| qc | Extract control probe intensities, compute QC metrics and generate QC reports. |
| genotype | Call genotypes, single nucleotide variants, and various related file conversions. |
| pgx | Pharmacogenomics CNV, star allele calling and metabolizer status annotation. |
| cyto | Cytogenetics CNV/LOH/mosaic calling and annotation. |
| help | Display more information on a specific command. |
| version | Display version information. |
refcli help— "Displays the first-layer help information." (L:160)refcli version— "Displays current [product] Local version." (L:164). Used as installation check (L:55).- Every module and sub-module also has its own
helpandversioncommands; every leaf command has--helpand (almost all)--versionflags.
Full command tree:
refcli
├── help
├── version
├── qc
│ ├── call
│ ├── report
│ ├── help
│ └── version
├── genotype
│ ├── call
│ ├── gtc-to-bedgraph
│ ├── gtc-to-vcf
│ ├── help
│ └── version
├── pgx
│ ├── copy-number
│ │ ├── call
│ │ ├── train
│ │ ├── help
│ │ └── version
│ ├── star-allele
│ │ ├── call
│ │ ├── annotate
│ │ ├── help
│ │ └── version
│ ├── help
│ └── version
└── cyto (no sub-module)
├── call
├── annotate
├── help
└── version
1.2 Common option semantics (apply to all leaf commands where present)
| Flag | Type | Default | Meaning |
|---|---|---|---|
--debug |
boolean switch | false | "Includes stack traces in logs." / "Logs will include stack traces." |
--json-log |
boolean switch | false | "Outputs logs in JSON format." (JSON schema not documented on these pages) |
--help |
switch | — | Command help screen ("Display this help screen.") |
--version |
switch | — | "Displays version information." (absent from qc call table) |
--output-folder |
path | CWD for qc call, qc report, pgx copy-number train, star-allele call/annotate, cyto call/annotate; not stated for genotype call, gtc-to-bedgraph, gtc-to-vcf, pgx copy-number call | Output directory. Re-using an output folder overwrites older files (L:87). |
--sample-sheet |
path (CSV) | — | Filter samples and supply metadata; see §3.9 |
--idat-folder / --gtc-folder |
directory | — | Root folder; recursive (includes subdirectories, stated for idat-folder). Overrides sample-sheet RootFolder. |
Rule repeated for every sample-set command: "Either --idat-folder, --sample-sheet, or both are required inputs."
(qc call L:192, genotype call L:256) / "Either --gtc-folder, --sample-sheet, or both are required inputs."
(gtc-to-bedgraph L:273, gtc-to-vcf L:324). For pgx copy-number call/train and cyto call the same pattern is implied
(--gtc-folder not marked required) but not stated explicitly.
1.3 qc call (L:177-192)
"Performs QC sample analysis given raw iScan data (IDAT)." / "Extracts raw control probe intensities and computes QC metrics from IDAT files."
| Option | Req | Type | Default | Allowed | Description |
|---|---|---|---|---|---|
--array-type |
Required | enum | — | genotyping, methylation |
Array type for QC analysis |
--csv-manifest |
Required | path | — | — | CSV manifest |
--idat-folder |
"[Required]" in table, but see note | dir | — | — | IDAT dir incl. all subdirectories; overrides sheet RootFolder |
--sample-sheet |
optional | path | — | — | Sample sheet for filtering |
--output-folder |
optional | dir | CWD | — | Output dir |
--debug |
optional | bool | false | — | stack traces |
--help |
— | — | — | — | help |
--json-log |
optional | bool | false | — | JSON logs |
Note: "Either --idat-folder, --sample-sheet, or both are required inputs." [DOC-INCONSISTENCY] table says
--idat-folder "[Required]" while the note says sheet alone suffices. No --version, no --num-threads, no BPM
(QC uses the CSV manifest's [Controls] section, not BPM). Outputs (from L:106-107,196): controls.raw_metrics.csv,
controls.qc_metrics.csv.
1.4 qc report (L:194-216)
Generates QC Report (interactive HTML) + QC Table (spreadsheet) from one or more dataset folders.
Inputs per dataset folder: controls.raw_metrics.csv (required), controls.qc_metrics.csv (required), from
qc call or genotype call; gt_sample_summary.csv ("highly recommended", optional) from genotype call for
"functional QC". Supports "parent-folder discovery, dataset labeling, and interactive dashboards".
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--data |
Required | <path[,path,...]> |
— | One or more dataset folders, comma-separated; or one parent folder → datasets auto-discovered beneath it |
--config |
optional | <file> YAML |
built-ins | Default QC thresholds. CLI thresholds override config. Built-in defaults callRate = 0.98, logRDev = 0.20; "All other thresholds are unset by default." |
--label |
optional | <label[,label,...]> |
dataset folder names | One label per dataset; count must equal number of datasets |
--output-folder |
optional | dir | CWD | |
--output-format |
optional | <csv|xlsx> |
xlsx |
Format of sample QC table; XLSX has conditional colouring and a "Thresholds" sheet |
--threshold-callRate |
optional | <value> float |
0.98 |
Functional QC call-rate threshold; overrides YAML |
--threshold-logRdev |
optional | <value> float |
0.20 |
Functional QC LogRDev threshold; overrides YAML |
--debug |
optional | bool | false |
|
--json-log |
optional | bool | false |
|
--help, --version |
— | — | — |
Precedence: CLI threshold > YAML --config > built-in defaults (L:212).
Example (verbatim, L:109): refcli qc report --data /user/gtc --config /user/config.yaml --output-folder /user/qc_report --label dataset_name
Privacy warning (danger hint, L:214-216, verbatim): "If the input data for qc report was generated while using a
sample sheet, sample metadata such as sample names, process information, and other sample-sheet values will be
carried into the QC report outputs, including tables and HTML files. Review QC report outputs before sharing them.
If you need to remove some metadata before sharing, remove it from gt_sample_summary.csv and rerun the qc report
command." → i.e. the report reads metadata from gt_sample_summary.csv, and editing that CSV is the supported
redaction path. Also covers methylation arrays (L:101). Details delegated to sub-page qc-report.md (anchors:
#metadata-propagation-gt_sample_summary, #multiple-dataset-folders,
#parent-folder-containing-multiple-dataset-folders, #configuration-file-optional).
1.5 genotype call (L:238-256)
"Determines genotype calls (GTC) from IDAT files."
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--bpm-manifest |
Required | path | — | BPM manifest |
--cluster-file |
Required | path | — | EGT cluster file |
--idat-folder |
cond. | dir | — | recursive; overrides sheet RootFolder |
--sample-sheet |
cond. | path | — | filtering + metadata |
--debug |
opt | bool | false | |
--gencall-cutoff |
opt | float | 0.15 | "GenCall score cutoff to label a NoCall." |
--help |
— | |||
--json-log |
opt | bool | false | |
--num-threads |
opt | int | not documented | "Number of parallel threads to run." (only command with a threads flag) |
--output-folder |
opt | dir | not stated | |
--version |
— |
Outputs (implied on this page): <sample>.gtc (GTC v5, L:277), controls.raw_metrics.csv,
controls.qc_metrics.csv, gt_sample_summary.csv (genotype sample summary; sample-sheet optional columns are
echoed as-is, I:104).
Example (verbatim): refcli genotype call --bpm-manifest /user/productfiles/manifest.bpm --cluster-file /user/productfiles/clusterfile.egt --idat-folder /user/IDATs --output-folder /user/gtc
1.6 genotype gtc-to-bedgraph (L:258-273)
Converts GTC to BedGraph visualisation files "from the Log R Ratio and B-allele frequency data" (L:260); the module table (L:233) says only "log R ratio data" [DOC-INCONSISTENCY].
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--bpm-manifest |
Required | path | — | BPM |
--gtc-folder |
cond. | dir | — | overrides RootFolder |
--sample-sheet |
cond. | path | — | |
--debug, --json-log |
opt | bool | false | |
--help, --version |
— | |||
--output-folder |
opt | dir | not stated |
1.7 genotype gtc-to-vcf (L:275-325)
"Converts GTC (v5) to SNV VCF Files. The command is only applicable for Genotype Call Files produced by [product] Array." (i.e. GTCs from other producers, e.g. older AutoConvert/GenomeStudio, are unsupported/untested).
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--bpm-manifest |
Required | path | — | BPM |
--csv-manifest |
Required | path | — | CSV manifest "with SourceSeq column" |
--genome-fasta-file |
Required | path (.fa) | — | ".fai" must be in same directory |
--gtc-folder |
cond. | dir | — | overrides RootFolder |
--sample-sheet |
cond. | path | — | |
--auxiliary-loci |
opt | path (VCF) | — | "VCF file with auxiliary definitions of loci, such as for multi-nucleotide variants" |
--debug |
opt | bool | false | |
--disable-genome-cache |
opt | bool | (cache on) | "Disables the reference genome cache." |
--filter-loci |
opt | (path? text file) | — | "Generates a text file containing a list of probe names to be filtered." [DOC-INCONSISTENCY/ambiguous]: wording says "generates", but it is almost certainly an input list of probe names to exclude (L:419 "without filter loci" treats it as a filter applied to VCF content). |
--unsquash-duplicates |
opt | bool | false | "Generates unique VCF records for duplicate assays." |
--help |
— | |||
--json-log |
opt | bool | false | |
--no-bgzip |
opt | bool | false | "VCFs are not bgzip compressed (.gz) and no tabix index files (.tbi) are output." |
--use-infI-nc-info |
opt | bool | false | "When combining loci, use Infinium I probe no-calls to constrain possible genotypes." (mixed-case I in flag) |
--output-folder |
opt | dir | not stated | |
--version |
— |
Example (verbatim): refcli genotype gtc-to-vcf --bpm-manifest /user/productfiles/manifest.bpm --csv-manifest /user/productfiles/manifest.csv --genome-fasta-file /user/productfiles/genome.fa --gtc-folder /user/gtc --output-folder /user/vcf
Output file naming (from I:181 / L:425): *.snv.vcf.gz (+ .tbi), or .snv.vcf with --no-bgzip.
Sub-behaviours: squashing duplicates, genome cache, auxiliary loci — see §2.
1.8 pgx copy-number call (L:357-371)
"Determines copy number variants given genotypes (GTC to CNV VCF)." "A batch of 24 samples or more are required for analysis. For a successful analysis, 22 samples must pass QC defined as having log R dev < 0.2."
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--cn-model |
Required | path (.dat) | — | PGx CN model |
--gtc-folder |
cond. | dir | — | overrides RootFolder |
--sample-sheet |
cond. | path | — | |
--debug, --json-log |
opt | bool | false | |
--no-bgzip |
opt | bool | false | as above |
--output-folder |
"[Optional]" | dir | not stated | Recommended: same folder as SNV VCFs (star-allele call takes one VCF folder) |
--help, --version |
— |
Note: no BPM/FASTA/EGT argument — the CN model must therefore embed/locate everything else needed (and is matched to
the manifest revision; mismatch → error, I:37). Output: *.cnv.vcf.gz.
Example (verbatim): refcli pgx copy-number call --cn-model /user/productfiles/manifest_pgx_model.dat --gtc-folder /user/gtc --output-folder /user/vcf
1.9 pgx copy-number train (L:373-396)
"Trains copy number model for a set of samples (GTC to CN Model File)." Generate a new model when using a customised EGT.
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--bpm-manifest |
Required | path | — | BPM; mask file (.msk) assumed in same directory |
--genome-fasta-file |
Required | path | — | .fai in same directory |
--platform |
Required | enum | — | LCG for GDA-ePGx; EX for GSAv4-ePGx or GCRA-ePGx. Value = Assay Format heading of the CSV manifest (L:587) |
--gtc-folder |
cond. | dir | — | overrides RootFolder |
--sample-sheet |
cond. | path | — | |
--debug, --json-log |
opt | bool | false | |
--disable-genome-cache |
opt | bool | (cache on) | |
--help, --version |
— | |||
--output-folder |
opt | dir | CWD | "The location to output the CN model." |
Example (verbatim, L:127): refcli pgx copy-number train --bpm-manifest /user/productfiles/manifest.bpm --genome-fasta-file /user/productfiles/genome.fa --gtc-folder /user/gtc --platform LCG --output-folder /user/productfiles/cnmodelnew
Second example (L:588, verbatim, [DOC-INCONSISTENCY] missing pgx module): refcli copy-number train --bpm-manifest /user/productfiles/manifest.bpm --genome-fasta-file /user/productfiles/genome.fa --gtc-folder /user/new_gtcs --platform LCG --output-folder /user/productfiles/new_cnmodel
1.10 pgx star-allele call (L:417-431)
"Calls PGx star allele diplotypes. The SNV VCF files should be generated using the [product] gtc-to-vcf command with unsquash-duplicates off (default) and without filter loci."
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--database |
Required | path (.zip) | — | PGx database |
--license-server-url |
Required | URL w/ credentials | — | e.g. https://username:password@[ref-license-server] |
--vcf-folder |
Required | dir | — | contains *.snv.vcf.gz and *.cnv.vcf.gz |
--query-license-quota |
opt | bool | — | query and display quotas at beginning and end |
--debug, --json-log |
opt | bool | false | |
--output-folder |
opt | dir | CWD | |
--help, --version |
— |
No --sample-sheet (samples come from VCF folder). Output: star_alleles.dat (name from the annotate example).
Example (verbatim): refcli pgx star-allele call --vcf-folder /user/vcf --database /user/productfiles/DAv1.4.0-rc2-pgx-mapping-sha.b164b4aa02f34cc9b72aba020c3a8735cb8e4421.zip --output-folder /user/star-alleles --license-server-url https://username:password@[ref-license-server]
Note: database file name embeds software version + content SHA (<ver>-pgx-mapping-sha.<40-hex>.zip).
1.11 pgx star-allele annotate (L:433-445)
"Annotates and summarizes the star-alleles, specifically for metabolizer statuses and outputs in a consolidated JSON report. Metabolizer status is determined through direct lookup into public PGx guidelines CPIC or DPWG." Quick-start text says it also summarizes into CSV files (L:124).
| Option | Req | Type | Default | Allowed | Description |
|---|---|---|---|---|---|
--star-alleles |
Required | path (.dat) | — | — | from star-allele call |
--guidelines |
opt | enum | CPIC |
CPIC, DPWG |
guideline set |
--debug, --json-log |
opt | bool | false | ||
--output-folder |
opt | dir | CWD | ||
--help, --version |
— |
Example (verbatim): refcli pgx star-allele annotate --star-alleles star_alleles.dat --guidelines CPIC --output-folder /user/metabolizer-statuses
No license needed for annotate (only star-allele call requires quota).
1.12 cyto call (L:466-508)
"Determines copy number variants (CNV) and loss/absence of heterozygosity (LOH/AOH) given genotypes." Also mosaic events.
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--cn-model |
Required | path (.dat) | — | cyto model |
--gtc-folder |
cond. | dir | — | overrides RootFolder |
--sample-sheet |
cond. | path | — | |
--debug, --json-log |
opt | bool | false | |
--no-bgzip |
opt | bool | false | (must NOT be used if cyto annotate follows) |
--output-folder |
"[Optional]" | dir | CWD | |
--help, --version |
— | |||
--min-del-probes |
opt | int | 10 | deletions reported only if ≥ N probes |
--min-del-size |
opt | kb | 25 | deletions ≥ size |
--min-dup-probes |
opt | int | 10 | |
--min-dup-size |
opt | kb | 50 | |
--min-loh-probes |
opt | int | 500 | |
--min-loh-size |
opt | kb | 3000 | |
--min-gainloh-probes |
opt | int | 300 | GAINLOH events |
--min-gainloh-size |
opt | kb | 500 | |
--min-mdel-probes |
opt | int | 10 | mosaic deletions |
--min-mdel-size |
opt | kb | 25 | |
--min-mdup-probes |
opt | int | 10 | mosaic duplications |
--min-mdup-size |
opt | kb | 50 | |
--min-mloh-probes |
opt | int | 500 | mosaic LOH |
--min-mloh-size |
opt | kb | 3000 | |
--max-mosaic-fraction |
opt | float | 1.0 | mosaic variants with fraction ≥ this are "promoted", marked INFO HIGHFRACTION (replaces MOSAIC tag), retain their mosaic fraction value |
--smoothing |
opt | int | 5 | LRR smoothing window: number of probes on each side of centre probe |
Filter descriptions all say "will be reported in the json file" — i.e. the thresholds on cyto call govern what is
reported downstream (the cyto call VCF and/or a cyto.cnv.dat metadata file), duplicated on cyto annotate.
Example (verbatim): refcli cyto call --cn-model /user/productfiles/cyto_model.dat --gtc-folder /user/gtc --output-folder /user/vcf
1.13 cyto annotate (L:510-545)
"Annotates samples and generates cytogenetic json reports." (gene annotations, cytogenetic bands, QC fields, variant info from the VCFs, L:138)
| Option | Req | Type | Default | Description |
|---|---|---|---|---|
--annotation-db |
Required | path (.zip) | — | cyto annotation database |
--vcf-folder |
Required | dir | — | contains *.cnv.vcf.gz (+ cyto.cnv.dat) |
--output-folder |
"[Optional]" | dir | CWD | |
--debug, --json-log |
opt | bool | false | |
--help, --version |
— | |||
--min-del-probes … --min-mloh-size |
opt | same 12 flags and defaults as cyto call | ||
--min-qual |
opt | number | 20 | "Min CNV qual and LOH qual scores." |
(no --max-mosaic-fraction, --smoothing, --no-bgzip, --sample-sheet on annotate)
Example (verbatim): refcli cyto annotate --annotation-db /user/productfiles/CytoAnnotateData_DAv1.2.0.zip --vcf-folder user/vcf --output-folder /user/cyto-annotations
(note the relative user/vcf in the original — likely a typo.)
1.14 Mutually exclusive / dependent option groups
- At-least-one-of groups: {
--idat-folder,--sample-sheet} (qc call, genotype call); {--gtc-folder,--sample-sheet} (gtc-to-bedgraph, gtc-to-vcf, implicitly pgx cn call/train, cyto call). When both given, folder option overrides sheetRootFolder. - Threshold precedence:
--threshold-*>--configYAML > built-in. --labelcardinality must equal number of--datadatasets.- Pipeline incompatibilities:
--unsquash-duplicatesand--filter-locimust not be used when star-allele call follows;--no-bgzipon cyto call must not be used if cyto annotate follows. - No true mutually-exclusive flags are documented.
1.15 Quick-start workflows (verbatim command sequences)
Genotyping: genotype call → genotype gtc-to-vcf. QC report: genotype call (or qc call) → qc report. PGx (L:113-127): genotype call → gtc-to-vcf → pgx copy-number call (same output folder as SNV VCF) → pgx star-allele call → pgx star-allele annotate → [optional] pgx copy-number train. Cytogenetics (L:129-139): genotype call → cyto call → cyto annotate. Semi-custom PGx (L:614-623): GenomeStudio custom EGT → genotype call (semi-custom BPM/EGT) → gtc-to-vcf (semi-custom BPM/CSV) → full commercial-product PGx pipeline separately.
refcli genotype call --bpm-manifest /user/productfiles/semi_custom_manifest.bpm --cluster-file /user/productfiles/semi_custom_clusterfile.egt --idat-folder /user/IDATs --output-folder /user/semi_custom_gtcs
refcli genotype gtc-to-vcf --bpm-manifest /user/productfiles/semi_custom_manifest.bpm --csv-manifest /user/productfiles/semi_custom_manifest.csv --genome-fasta-file /user/productfiles/genome.fa --gtc-folder /user/semi_custom_gtcs --output-folder /user/semi_custom_vcfs
refcli genotype call --bpm-manifest /user/productfiles/manifest.bpm --cluster-file /user/productfiles/new_clusterfile.egt --idat-folder /user/IDATs --output-folder /user/new_gtcs
2. Documented behaviours
2.1 Computing requirements (L:13-24)
| Category | Recommendation |
|---|---|
| CPU | 8 cores |
| Memory | 32 GB available or more |
| Disk | 30 GB or more free |
| OS | Windows 10+ (win10-x64); CentOS 7+, Ubuntu 20.04+ (linux-x64) |
- Cybersecurity: must not require admin; "We recommend you do not run with elevated permissions."
- Up to 500 GDA-ePGx samples per analysis batch on this hardware (L:71).
cyto annotateneeds ≥ 5 GB free disk (L:545).
2.2 Installation (L:42-65)
- Download platform package (Windows/Linux) from support site; move to desired folder (admin may be needed for
/usr/local/bin,C:\Program Files). - Unzip; executable is in the
refclisubfolder, e.g./usr/local/bin/<refcli>-linux-x64-DAv1.4.0/<refcli>/. - Optionally add to PATH; verify with
refcli version(prints version on success).
- Distributed as a self-contained .NET application (bundled runtime) but needs host native libraries (the .NET DEB/RPM dependency lists: ICU, OpenSSL, zlib, libstdc++, etc.); missing libs → fails to start or runtime errors.
- Package naming convention:
<refcli>-<rid>-DAv<version>(rid = win10-x64 / linux-x64).
2.3 Licensing / quota (L:26-40) — only pgx star-allele call
- Quota charged per sample analysed, including re-analysis and low-quality samples.
- Quota checked "before and after analysis but not after updating the usage"; re-run to see current usage.
- Credential from activation email passed via
--license-server-url; logs record remaining quota at beginning and end. - Internet required. Endpoints: v1.0/v1.1
[ref-license-server-legacy]; v1.2+[ref-license-server](new domain does not work with v1.0/1.1). - v1.1+: precomputed quota no longer checked during analysis → an over-quota run proceeds and fails only at the end:
Error updating usage. HTTP error status code: 409, and results are not written to disk. (Irrelevant to an open reimplementation except as a UX anti-pattern: never discard computed results.)
2.4 Sample minimums / batch rules
- Genotyping and cyto: no sample minimum (L:69).
- PGx CNV call: ≥ 24 samples, of which ≥ 22 pass QC (LogRDev < 0.2) (L:71, L:359). Recommended ≥ 96 samples passing LogRDev ≤ 0.2 (L:121); "Analyze batches of 96 samples or more" (L:77).
- PGx CN train: ≥ 96 samples, ≥ 90 passing LogRDev ≤ 0.2; recommended ≥ 150; diminishing returns and longer runtime beyond 3,000 (L:379, L:598).
- Batch recommendations for PGx CNV (L:73-79): same assay batch (WGA + targeted gene amplification) → same analysis batch; don't combine different reagent lots; samples processed within a two-week period from multiple library prep batches may be pooled (same reagent lots/instruments); use the standard CN model and PGx DB.
- Note boundary inconsistencies: "< 0.2" (call) vs "≤ 0.2" (train, recommendation) [DOC-INCONSISTENCY].
2.5 File discovery
--idat-folderis searched recursively ("This path also includes the contents of all subdirectories").- Duplicate IDAT pairs for the same
SentrixBarcode_Positionunder one--idat-folderare not allowed (rescans must be removed/relocated beforegenotype call/qc call) (L:565). Behaviour on violation not stated (presumably error). - Samples identified by
<12-digit barcode>_<RxxCxx>_{Grn,Red}.idat(I:12). - Sample sheet selects a subset of samples from the folder ("A sample sheet can be used to select specific samples out of a folder", L:85).
RootFolderin[Header](absolute path) is used when no folder option; CLI folder option wins on conflict.- star-allele call pairs
*.snv.vcf.gzand*.cnv.vcf.gzin one folder; cyto annotate needs*.cnv.vcf.gz+ thecyto.cnv.datmetadata file in the same folder. - CN train finds
.msknext to the BPM with the same base name as the manifest; FASTA commands find.fainext to the.fa.
2.6 Overwrite behaviour
"[product] will overwrite older files if using the same --output-folder from a previous analysis. If this is not
desired, use different --output-folder for re-analyses." (L:87) — silent overwrite, no --force.
2.7 Threading
Only genotype call --num-threads ("Number of parallel threads to run."), default undocumented. Other commands have
no thread control.
2.8 Genome cache (L:302-304)
Default: whole reference genome read into memory (faster than indexed disk reads, more RAM).
--disable-genome-cache for low memory or small manifests (gtc-to-vcf, pgx copy-number train).
2.9 Logging / exit
--debug: include stack traces in logs (default off).--json-log: JSON-formatted logs (default off). Field schema not documented here (see output-files page#section-warningerror-messages-and-logs).- Logs record license quota at start/end for star-allele call.
- Cancel with Ctrl+C. Exit codes not documented. On quota failure, nothing written.
- Error on cluster/CN-model mismatch in v1.0: "produce an error and exit" (L:594).
2.10 Squashing duplicates (L:298-300)
Default: multiple assays probing the same variant (same or alternate designs) are squashed into one VCF record
reflecting the true variant; method in output-files VCF description. --unsquash-duplicates → one record per assay
(for assay validation). Exception: multi-allelic loci queried with duplicated designs are never unsquashed.
"DO NOT use --unsquash-duplicates option if doing star allele calling downstream as that command expects squashed
variants."
2.11 Infinium I no-call info
--use-infI-nc-info: when combining loci, Infinium I probe no-calls constrain possible genotypes (details on
output-files page #using-infinium-i-nocall-information). Default off.
2.12 Auxiliary loci (L:306-320)
MNVs unsupported upstream; can be designed as SNP assay e.g. true ATGC[AT/CG]GTAA designed as ATGC[A/C]NNNN.
--auxiliary-loci <VCF> restores true alleles. Restrictions (verbatim):
- "The auxiliary definition must NOT be a multi-allelic variant."
- "The auxiliary definition must be a multi-nucleotide variant."
- "There must NOT be multiple array assays (e.g., duplicates) for the locus."
2.13 Cyto call behaviours (L:498-508)
-
10 probes per region recommended for confident CNV calls.
- Any probe-min threshold < 10 is silently reset to 10 (applies to call and annotate).
- Threshold > effective genome size → all variants filtered out.
-
10 events (DEL/DUP/AOH) per chromosome → indication for visual inspection.
- Mosaic fraction not estimable (insufficient informative probes) →
NaN. - Mosaic events ≥
--max-mosaic-fraction: INFOMOSAICreplaced byHIGHFRACTION. - Mosaic events < 15–20 % fraction may be missed (sensitivity limit).
- LogRDev > 0.2 → low-quality; flagged with
SampleQualityflag in VCF. - LogRDev > 0.3 → very noisy; only whole-chromosome-level events detected/reported.
2.14 Cyto annotate behaviours (L:539-545)
cyto.cnv.datmetadata file produced by cyto call must remain in the vcf-folder.- VCFs must be bgzipped + indexed.
- ≥ 5 GB free disk.
2.15 Custom cluster files and CN models (L:572-602)
- Custom EGT created in GenomeStudio 2.0; EGT file name: only
[A-Za-z0-9_-], no spaces/special chars. - For PGx products a GenomeStudio auxiliary file (tab-delimited, Column Import) maps Infinium assays → PGx star alleles, for reviewing cluster positions of PGx variants.
- If EGT customised, PGx CN model should be retrained with the same samples used for the EGT.
- v1.1+: CN model with a different cluster file → warning, proceeds. v1.0: same EGT required → error & exit.
- Training QC recommendation: remove samples with LogRDev > 0.2, call rate < 0.99, or TGA control probe < 1.0; same reagent lots; few batches.
- Algorithm assumption: training cohort's CN distribution resembles standard population distributions.
- Validate a retrained CN model with truth data before use.
2.16 Semi-custom arrays (L:604-625)
- PGx CNV and star-allele calls limited to commercial content; add-on content excluded.
- Retain all backbone content when designing semi-custom PGx arrays.
- Run pipeline twice: semi-custom files for SNV VCFs, commercial files for PGx. Semi-custom GTC/VCFs "cannot be used for downstream PGx analysis commands". Keep in clearly labelled folders.
2.17 CLI tips (L:561-570)
Quote paths with spaces; case-sensitive options; no duplicate IDAT pairs; up-arrow to edit; copy-paste paths; Windows "Copy as Path"; Ctrl+C cancels.
3. Input file types
| # | File | Ext | Source | Commands (option) |
|---|---|---|---|---|
| 3.1 | IDAT | .idat | scanner (iScan, NextSeq550 for select arrays) | genotype call, qc call (--idat-folder) |
| 3.2 | CSV manifest | .csv | vendor | gtc-to-vcf, qc call (--csv-manifest) |
| 3.3 | BPM manifest | .bpm | vendor | cn train, genotype call, gtc-to-bedgraph, gtc-to-vcf (--bpm-manifest) |
| 3.4 | Cluster file | .egt | vendor or GenomeStudio | genotype call (--cluster-file) |
| 3.5 | PGx CN model | .dat | vendor or pgx copy-number train |
pgx cn call (--cn-model) |
| 3.6 | Cyto CN model | .dat | vendor | cyto call (--cn-model) |
| 3.7 | Mask | .msk | vendor | cn train (implicit, next to BPM) |
| 3.8 | PGx database | .zip | vendor | star-allele call (--database) |
| 3.9 | Cyto database | .zip | vendor | cyto annotate (--annotation-db) |
| 3.10 | Genome FASTA (+.fai) | .fa | vendor | gtc-to-vcf, cn train (--genome-fasta-file) |
| 3.11 | YAML config | .yaml/.yml | template or user | qc report (--config) |
| 3.12 | Sample sheet | .csv | user | genotype call, gtc-to-bedgraph, gtc-to-vcf, cn call, cn train, cyto call, qc call (--sample-sheet) |
| 3.13 | GTC | .gtc | genotype call | gtc-to-bedgraph, gtc-to-vcf, cn call, cn train (--gtc-folder) [cyto call also] |
| 3.14 | SNV VCF / PGx CNV VCF | .snv.vcf / .cnv.vcf | gtc-to-vcf / cn call | star-allele call (--vcf-folder) |
| 3.15 | PGx DAT | .dat | star-allele call | star-allele annotate (--star-alleles) |
| 3.16 | Cyto CNV VCF | .cnv.vcf | cyto call | cyto annotate (--vcf-folder) |
| 3.17 | QC dataset folder CSVs | .csv | qc call / genotype call | qc report (--data) |
| 3.18 | Auxiliary loci VCF | .vcf | user | gtc-to-vcf (--auxiliary-loci) |
| 3.19 | Filter-loci list | .txt | user | gtc-to-vcf (--filter-loci) |
Note: the summary table omits cyto call from GTC consumers and omits 3.17–3.19.
Product files (all but IDATs) download: [ref-docs:downloads].
Details:
- 3.1 IDAT: one pair per sample, Red and Grn channels, intensities per probe. Identified by 12-digit Sentrix
barcode (e.g.
123456789101), positionR01C01, andGrn/Red. - 3.2/3.3 Manifests: commercial from support site; custom/semi-custom from MyIllumina. "[product] only supports
manifest files from the Illumina Support site." CSV is human-readable complement to BPM; required for indel VCF
generation; gtc-to-vcf may be inaccurate if mapping info wrong. Mapping follows implicit dbSNP standard:
1-based positions; PAR positions reported on X; insertion → position of base immediately 5′ of insertion
(plus strand); deletion → position of most 5′ deleted base (plus strand). CSV heading contains
Assay Format(e.g.LCG,EX) used for--platform. CSV must haveSourceSeq. - 3.4 EGT: required for genotype call; custom may be needed; naming restriction
[A-Za-z0-9_-]. - 3.5 PGx CN model: one per PGx product; named after the manifest revision (e.g. B1) and must be used with the same-revision manifest. Must be retrained when EGT customised (v1.1+ only warns on mismatch).
- 3.6 Cyto model: one per supported product; named after manifest revision; must match. Contains primarily probe GC content information; tied to probe sequence/manifest; no retraining needed unless manifest or software version changes; independent of cluster file. Custom products: contact support with BPM, EGT and IDAT/GTC for ≥ 1 sample.
- Model/manifest mismatch → error during
pgx copy-number callandcyto call(probes added/removed between revisions). - 3.7 Mask: required for CN train; same base name as manifest, same folder as BPM; one per PGx product.
- 3.8 PGx DB: zip; each line maps one probe ID → a variant HGVS tag (many probes → one variant); cross-referenced with SNV VCF IDs at runtime; one DB works across all supported PGx products.
- 3.9 Cyto DB: zip of Ensembl + RefSeq data; product-independent; may be updated.
- 3.10 FASTA: .fa + .fai in same folder; PGx supports GRCh37 and GRCh38; custom references: contig IDs must match
manifest chromosome IDs exactly —
>1, not>chr1, for standard human manifests. - 3.11 YAML: set threshold to a number to enable,
nullto disable; without--configbuilt-ins forcallRateandlogRDevonly; cloud uses chemistry-specific default configs. - 3.12 Sample sheet: see §3.12 detail below.
3.12 Sample sheet detail (I:56-147)
- CSV. Required key columns: local
SentrixBarcode_A,SentrixPosition_A; cloudbeadChipName,sampleSectionName. Backwards compatible with most GenomeStudio sheets. - Sections optional:
[Header](withRootFolder,<abs path>, arbitrary keys e.g.Date,1/1/2025),[Data].[Manifests]section is ignored. Sectionless sheets valid (header row first line). - Valid examples (verbatim):
SentrixBarcode_A,SentrixPosition_A
204753010023,R02C01
SentrixBarcode_A,SentrixPosition_A,Sample_ID,Sample_Group,MetaData1
204753010023,R01C01,NA1231,Group1,F
204753010024,R01C01,NA1233,Group2,M
[Header]
RootFolder,/tests/samples
Date,1/1/2025
[Data]
SentrixBarcode_A,SentrixPosition_A,Sample_ID,Sample_Group,MetaData1
204753010023,R01C01,NA1231,Group1,F
204753010024,R01C01,NA1233,Group2,M
- Column names case-insensitive; duplicates after case-folding are an error.
- Column names must not collide with genotype-summary output fields (reserved keywords, e.g.
Sex Estimate). - Optional columns are echoed as-is into genotype summary files (
gt_sample_summary.csv). - Column names and field values ≤ 500 characters.
- Excel: format
SentrixBarcode_Aas number with 0 decimals; text-with-decimal or scientific notation → IDAT names cannot be resolved → run fails. - Sample name precedence: (1) sheet
Sample_Name(if column present, must be non-empty in every row and must not conflict with default IDSentrixBarcode_Position, e.g.204753010023_R01C01); (2) sample name embedded in IDAT/GTC metadata; (3)SentrixBarcode_Position. Sheet value never rewrites the name stored in the GTC; affects only downstream outputs (summary, VCFs); "the samplesheet for any given command is treated as the source of truth". - Cloud methylation QC sheet: no sections; keys
beadChipName,sampleSectionName;Sample_Groupcolours PCA control plot/Principal Component Summary; any extra columns after Sample_Group replicated to Sample QC Summary outputs.
beadChipName,sampleSectionName,Sample_ID,Sample_Group,MetaData1
204753010023,R01C01,NA1231,Group1,F
204753010023,R02C01,NA1232,Group2,F
204753010024,R01C01,NA1233,Group2,M
204753010024,R02C01,NA1234,Group1,M
- Cloud cyto + interpretation sheet: column
demographicSex∈ {M,F,U} compared againstSex Estimate.
SentrixBarcode_A,SentrixPosition_A,demographicSex
204753010023,R01C01,F
204753010023,R02C01,F
204753010024,R01C01,M
204753010024,R02C01,M
4. Error / warning messages (verbatim) and triggers
| Message | Trigger | Source |
|---|---|---|
Error updating usage. HTTP error status code: 409 |
v1.1+ star-allele call exceeded quota; detected at end; results not written | L:40 |
Duplicate column sample_name found. Column names are case-insensitive. Please remove or rename the column from the samplesheet and re-process. |
two sheet columns equal after case-folding (e.g. Sample_Name, sample_name) |
I:102 |
Sex Estimate is a reserved keyword. Please remove or rename the column from the samplesheet and re-process. |
sheet column collides with a genotype-summary output field | I:103 |
| (warning, text not given) CN model built with different cluster file | v1.1+ pgx copy-number call; proceeds | L:593 |
| (error & exit, text not given) cluster file differs from CN-model training | v1.0 | L:594 |
| (error, text not given) CN/cyto model vs manifest mismatch | pgx copy-number call, cyto call | I:37 |
| (failure, text not given) barcode unreadable from sheet (Excel sci-notation) | IDAT name resolution | I:107 |
| (validation, text not given) name/value > 500 chars | sample sheet | I:106 |
(validation, implied) Sample_Name empty in some row, or equals another sample's default ID |
sample sheet | I:113 |
(implied) --label count ≠ dataset count |
qc report | L:202 |
(implied) missing controls.*_metrics.csv in dataset folder |
qc report | L:200 |
| (silent) probe threshold < 10 reset to 10 | cyto call/annotate | L:501, L:541 |
| (implicit) license domain mismatch with v1.0/1.1 | star-allele call | L:39 |
| "application may fail to start or produce runtime errors" | missing .NET native libs | L:61 |
5. Numeric thresholds and defaults (consolidated)
| Value | Meaning |
|---|---|
| 8 cores / 32 GB / 30 GB disk | recommended hardware |
| 5 GB | free disk for cyto annotate |
| 500 | max GDA-ePGx samples per batch on recommended HW |
| 0.15 | --gencall-cutoff default (GenCall score < cutoff → NoCall) |
| 0.98 | qc report call-rate threshold default |
| 0.20 | qc report LogRDev threshold default; PGx/cyto QC boundary |
| 0.3 | LogRDev above which cyto reports only whole-chromosome events |
| 0.99 | call-rate recommended minimum for PGx samples / CN training samples |
| 1.0 | TGA control probe metric recommended minimum (PGx review/training) |
| 24 / 22 | PGx CNV call minimum batch / minimum passing LogRDev < 0.2 |
| 96 | recommended PGx CNV batch; minimum CN train samples |
| 90 | CN train samples passing LogRDev ≤ 0.2 |
| 150 | recommended CN train size |
| 3,000 | CN train diminishing returns |
| 2 weeks | window for pooling library-prep batches |
| 10 / 25 kb | DEL probes / size |
| 10 / 50 kb | DUP probes / size |
| 500 / 3000 kb | LOH probes / size |
| 300 / 500 kb | GAINLOH probes / size |
| 10 / 25 kb | mosaic DEL |
| 10 / 50 kb | mosaic DUP |
| 500 / 3000 kb | mosaic LOH |
| 10 | hard floor for any probe-count threshold |
| 1.0 | --max-mosaic-fraction |
| 5 | --smoothing probes each side (window = 11 probes) |
| 20 | --min-qual (cyto annotate) |
| 15–20 % | mosaic detection lower limit |
| > 10 events/chromosome | visual-inspection trigger |
| 500 chars | sample-sheet name/value limit |
| 12 digits | Sentrix barcode length |
| 409 | HTTP status on over-quota |
| CPIC | default guideline |
| xlsx | default QC table format |
6. Algorithmic hints
- Genotype calling: GenCall score with no-call cutoff 0.15; uses BPM + EGT (standard GenCall-style pipeline). Outputs GTC v5 plus control metrics and sample summary (call rate, LogRDev, Sex Estimate, TGA control metric…).
- Control QC (
qc call): control-probe intensity extraction from IDAT using the CSV manifest (Controls section); supports genotyping and methylation arrays; outputs raw metrics and computed QC metrics. - Functional QC: pass/fail on call rate ≥ 0.98 and LogRDev ≤ 0.20 (direction presumed); other thresholds optional via YAML (numeric enables, null disables).
- gtc-to-vcf: REF from FASTA (whole-genome in-memory cache by default); indels resolved from CSV SourceSeq; duplicate
assays squashed into single variant-level genotypes; optional use of Infinium I no-calls as constraints; MNV
restoration from auxiliary VCF (SNP-designed proxies with
Nflanks); multi-allelic sites combine multiple assays; probe IDs used as VCF IDs (PGx DB joins on VCF ID). - BedGraph: LRR and BAF tracks from GTC + BPM positions.
- PGx CNV: batch-based caller (needs ≥ 24 samples/≥ 22 good ones → cohort normalisation / batch-effect sensitive); model file (.dat) tied to manifest revision and (in v1.0) to cluster file → model contains per-probe intensity reference parameters derived from EGT-normalised data; training requires BPM + mask (probe exclusion) + FASTA (likely GC content / probe positions) + platform code (LCG vs EX assay format), assumes population-typical CN distribution (i.e. unsupervised mixture fit where the modal state is CN2 and prior frequencies ≈ population).
- Star alleles: probe→HGVS mapping DB; SNV + CNV VCFs combined; expects squashed variants and unfiltered loci;
output
.datthen annotate with CPIC/DPWG lookup (direct table lookup for metabolizer status). - Cyto: model = probe GC content (→ GC-wave correction of LRR); LRR smoothing with ±5-probe window; CNV/LOH/AOH/
GAINLOH (gain with LOH, i.e. CN≥3 with no heterozygosity) and mosaic DEL/DUP/LOH with mosaic fraction estimate
(NaN if too few informative probes); promotion of high-fraction mosaic events (
HIGHFRACTION); QUAL scores for CNV and LOH (min 20 at annotate); sample-quality flag at LogRDev > 0.2; whole-chromosome-only mode at > 0.3; "effective genome size" concept; annotation with Ensembl/RefSeq genes and cytobands to JSON. - CN training QC: exclude LogRDev > 0.2, call rate < 0.99, TGA control < 1.0.
7. Gap analysis vs SimurgArray plan
Plan sections compared: §1 goals, §6 formats, §8 VCF, §9 QC, §10 CN/LOH, §11 PGx, §13 outputs, §14 CLI, §15, §19.
7.1 CLI structure & global behaviour
| # | Vendor behaviour | Plan status | Suggested spec addition |
|---|---|---|---|
| G1 | Hierarchical module sub-module command; help/version at every level |
flat simurg <verb> |
Keep flat verbs but add simurg <verb> --help everywhere and a simurg version subcommand; optionally provide a compatibility alias layer mapping genotype call, genotype gtc-to-vcf, qc call, qc report, cyto call … to simurg verbs for users migrating scripts. |
| G2 | --debug (stack traces) separate from logging format |
--log-level only |
Add --debug = include tracebacks + DEBUG level. |
| G3 | --json-log |
--log-json |
Fine; document JSON log schema (timestamp, level, sample, step, message, error_code). Consider accepting --json-log as alias. |
| G4 | Silent overwrite of output folder | "never silently overwrite (require --force)" | Keep safer default; document that re-running into the same folder requires --force or --resume. |
| G5 | Folder OR sheet OR both; folder overrides sheet RootFolder |
--idats DIR|--sheet F (exclusive) |
Allow both together: sheet filters, --idats/--gtcs overrides sheet RootFolder; sheet alone uses RootFolder; error if neither. |
| G6 | Recursive IDAT discovery; duplicate barcode_position pairs disallowed | recursive discovery mentioned | Specify: recursive search, .idat and .idat.gz, error E_DUP_IDAT listing all paths when the same barcode_position is found twice; `--allow-duplicates=first |
| G7 | --num-threads only on genotype call |
--threads common |
OK; specify default = all physical cores (document). |
| G8 | Genome in-memory cache default, --disable-genome-cache |
not specified | Add FASTA access policy: default load whole genome (or mmap) with --no-genome-cache to use indexed random access. |
| G9 | --no-bgzip on VCF writers |
always bgzip | Add --no-bgzip (plain .vcf, no index) with warning that downstream CN annotate/PGx require compressed+indexed input. |
| G10 | Exit codes undocumented; quota failure discards results | defined exit codes | Keep; add rule: results computed are always written before any post-processing step can fail. |
| G11 | Case-sensitive, mixed-case flags | n/a | Use lowercase kebab-case only; no mixed-case flags. |
| G12 | Self-contained distribution, OS list, HW recommendation (8 cores/32 GB/30 GB) | §3 dev machine only | Add "user runtime requirements" section: supported OS/arch, min/recommended RAM per workload (e.g. 500-sample CN batch), disk for annotation DBs, no-root operation. |
| G13 | Licence/quota server | none (open) | Out of scope; explicitly state no network access required at runtime (offline-first). |
7.2 Genotyping (genotype call equivalent)
| # | Vendor | Plan | Suggestion |
|---|---|---|---|
| G14 | Outputs per batch: controls.raw_metrics.csv, controls.qc_metrics.csv, gt_sample_summary.csv produced by genotype call itself |
§13 has controls_raw.tsv, controls_qc.tsv, sample_summary.tsv |
Make simurg genotype emit control metrics + sample summary automatically (so QC report works directly on the genotype output folder); offer CSV output with compatible column names as an option (--summary-format csv). |
| G15 | Sample-sheet metadata columns echoed as-is into sample summary | not specified | Specify: all non-key sheet columns propagated verbatim (order preserved) into sample_summary and QC report; reserved-name check. |
| G16 | GTC sample name not rewritten by sheet; sheet is source of truth per command | §6.6 precedence only | Add: name resolution happens per command; GTC-stored name is kept; downstream outputs use sheet value. Add rule Sample_Name non-empty for all rows if column present and must not equal another sample's default barcode_position ID. |
| G17 | Custom EGT naming restriction | none | Validate output EGT names from ArrayTrain [A-Za-z0-9_-] (portability), warn on others. |
7.3 gtc-to-vcf equivalent
| # | Vendor | Plan | Suggestion |
|---|---|---|---|
| G18 | --auxiliary-loci VCF to restore MNV alleles for SNP-proxy designs (with 3 restrictions) |
absent | Add --auxiliary-loci FILE.vcf: records keyed by probe ID/position override REF/ALT for single-assay, biallelic MNV loci; validate restrictions and error otherwise. |
| G19 | --filter-loci text list of probe names to exclude |
absent | Add --exclude-loci FILE (one probe name per line) and --include-loci; record in provenance; warn when PGx step later consumes filtered VCFs. |
| G20 | --use-infI-nc-info (Infinium I no-calls constrain combined genotypes) |
multi-probe merge in §8.3 but no Inf-I no-call rule | Add opt-in rule: when merging probes, an Infinium I probe no-call removes candidate genotypes that the probe would have been able to distinguish; default off. |
| G21 | Multi-allelic duplicated designs never unsquashed | §8.3 unsquash described generally | Specify exception: --unsquash-duplicates has no effect for multi-allelic sites. |
| G22 | Only GTC v5 from own caller accepted | plan reads v3–v5 | Keep broader support, but warn when GTC lacks v5 fields (BAF/LRR/percentiles) or was produced by other software. |
| G23 | CSV manifest required, must contain SourceSeq; mapping conventions (1-based, PAR on X, indel anchor positions) | §6.3/§8.1 partial | Add explicit validation of manifest mapping conventions: PAR loci on X coordinates, insertion anchor = base 5′ of insertion, deletion = first deleted base; convert to VCF left-anchored representation. |
| G24 | FASTA contig names must exactly match manifest chrom names (1 not chr1) |
alias table (more permissive) | Keep alias table but document default strict mode and the alias opt-in; verify .fai exists next to FASTA (or build it). |
| G25 | Output naming <sample>.snv.vcf.gz, CNV <sample>.cnv.vcf.gz |
<sample>.vcf.gz, <sample>.cnv.vcf.gz |
Adopt .snv.vcf.gz suffix to disambiguate SNV and CNV VCFs coexisting in one folder. |
7.4 bedGraph
| G26 | Dedicated command from GTC+BPM producing LRR and BAF bedGraphs | export bedgraph | Covered; ensure it takes GTC folder + sheet and works without re-running genotyping. |
7.5 QC
| # | Vendor | Plan | Suggestion |
|---|---|---|---|
| G27 | qc call works directly from IDAT + CSV manifest (no EGT/BPM), `--array-type genotyping |
methylation` | methylation QC is v2 optional; control metrics derived in genotype pipeline |
| G28 | qc report accepts multiple dataset folders (comma list) or a parent folder with auto-discovery, --label per dataset |
single report | Add multi-dataset report: --data DIR[,DIR...] or parent dir discovery (any subfolder containing the two control-metrics files); --label list with cardinality check; datasets shown side by side. |
| G29 | `--output-format csv | xlsx` (xlsx default with conditional colouring + "Thresholds" sheet) | TSV/XLSX |
| G30 | YAML config: numeric = enable, null = disable; only callRate/logRDev enabled by default; CLI > YAML > defaults |
§14 qc: block |
Adopt the same tri-level precedence and null-disables semantics; record effective thresholds with their source in outputs. |
| G31 | Privacy: sheet metadata flows into HTML; redact by editing the summary CSV and rerunning | none | Add --redact-columns COL,... and a warning in docs; report built only from summary file so edits propagate. |
| G32 | TGA control probe metric (< 1.0 flagged) used for PGx review | not in §9 | Add targeted-gene-amplification control metric for ePGx arrays to control QC (definition to be derived from control probes). |
7.6 PGx CNV and star alleles
| # | Vendor | Plan | Suggestion |
|---|---|---|---|
| G33 | CN call: ≥ 24 samples, ≥ 22 with LogRDev < 0.2, recommended ≥ 96 | §11 "≥ 24 samples" | Add explicit pre-flight check and error when fewer than N_min total / passing samples; configurable but defaulting to 24/22; warn below 96. |
| G34 | CN train: ≥ 96 samples, ≥ 90 passing; recommend ≥ 150; plateau ~3000; QC exclusions (LogRDev>0.2, CR<0.99, TGA<1.0) | "≥ 96 for training" | Add pass-count requirement and automatic exclusion rules with report of excluded samples. |
| G35 | Mask file (.msk) next to BPM with same base name, used in training | absent | Define an open probe-mask format (list of probe names excluded from CN modelling) discovered next to the manifest or given via --mask. |
| G36 | --platform (assay format LCG/EX) from CSV Assay Format heading |
absent | Read assay format from CSV heading automatically; allow override. |
| G37 | CN model tied to manifest revision (error on mismatch), and to cluster file (warn on mismatch) | absent | CN model file header must store manifest name/revision + probe-set hash and EGT hash; error on manifest mismatch, warning on EGT mismatch. |
| G38 | Batch recommendations (same reagent lot, 2-week window, same assay batch) | absent | Add documentation + optional batch-consistency warnings using sheet columns (e.g. plate/lot) if present. |
| G39 | Star-allele input folder with paired *.snv.vcf.gz/*.cnv.vcf.gz; requires squashed, unfiltered SNV VCFs |
§11 VCF input | Validate that VCF headers record unsquash=false and no filter list; refuse otherwise. |
| G40 | Probe→HGVS mapping DB (many probes → one variant), versioned zip with content hash in name | versioned SQLite pgx_db | Include a content hash + software-version in DB filename/metadata; record in provenance. |
| G41 | Separate annotate step: .dat → JSON + CSV, `--guidelines CPIC |
DPWG` default CPIC | "phenotype" subcommand |
| G42 | Semi-custom arrays: PGx limited to backbone content; run twice | none | Document how SimurgArray handles add-on content (likely we can include it natively; state policy). |
| G43 | Per-sample batch size limit (500 GDA samples at 32 GB) | performance plan | Add memory model for cohort CN steps and document max batch per RAM. |
7.7 Cytogenetics (genome-wide CNV/LOH)
| # | Vendor | Plan | Suggestion |
|---|---|---|---|
| G44 | Separate call (→ CNV VCF + metadata dat) and annotate (→ JSON) commands |
`cn call | train |
| G45 | Event classes include GAINLOH (300 probes / 500 kb) and mosaic DEL/DUP/LOH each with own thresholds | §10.2 has DEL/DUP/LOH/mosaic "same" | Add GAINLOH class and separate mosaic thresholds; 12 flags: --min-{del,dup,loh,gainloh,mdel,mdup,mloh}-{probes,size} with defaults 10/25kb, 10/50kb, 500/3000kb, 300/500kb, 10/25kb, 10/50kb, 500/3000kb. |
| G46 | Probe threshold floor = 10 (silently reset) | none | Enforce floor 10 with a logged warning (not silent). |
| G47 | --max-mosaic-fraction (default 1.0) → HIGHFRACTION INFO replaces MOSAIC |
none | Add parameter and INFO flags MOSAIC/HIGHFRACTION; MF=NaN when not estimable. |
| G48 | --smoothing ±5 probes LRR smoothing |
none | Add --smoothing K (probes each side, default 5) pre-segmentation smoothing. |
| G49 | --min-qual 20 at annotation |
QUAL defined, cap 60 | Add annotation-time QUAL filter default 20. |
| G50 | Sample-level SampleQuality VCF flag at LogRDev > 0.2; whole-chromosome-only above 0.3 |
§10.2 already has these | Covered; name the flag and header line. |
| G51 | >10 events per chromosome → needs review | none | Add per-sample QC flag EXCESS_EVENTS when any chromosome has >10 DEL/DUP/AOH calls. |
| G52 | Cyto model = probe GC content, per manifest revision, independent of EGT; custom model on request | §10.1 compute GC from FASTA | Good: we derive GC ourselves; persist as a per-manifest model file with manifest hash; no cluster dependency. |
| G53 | Cyto annotation DB (Ensembl+RefSeq zip), product-independent, JSON reports with genes, cytobands, QC fields | §10.4 | Specify JSON report schema and a versioned annotation bundle built by script; 5 GB disk note. |
| G54 | Annotate requires bgzipped + indexed VCF and sidecar in same folder | none | Validate and error clearly. |
| G55 | Mosaic sensitivity limit 15–20 % | §10.3 | Document target sensitivity; validate against it. |
7.8 Input formats
| # | Vendor | Plan | Suggestion |
|---|---|---|---|
| G56 | Sample sheet: 500-char limit; reserved names = output columns; duplicate case-insensitive columns error; [Manifests] ignored; Excel sci-notation barcode failure |
§6.6 partial | Add validations with messages: duplicate column (case-insensitive), reserved column name, >500 chars, empty Sample_Name, Sample_Name colliding with default IDs, non-integer barcode (detect 2.04753E+11 / 204753010023.0 and give a targeted hint). Consider auto-repairing .0 suffix with a warning. |
| G57 | [Header] RootFolder absolute path |
not in plan | Support RootFolder; relative paths resolved against sheet location (extension), CLI override. |
| G58 | Sectionless sheets valid (first line = header) | implied | Explicitly support. |
| G59 | Cloud variants: beadChipName/sampleSectionName keys; Sample_Group colouring in PCA plots; demographicSex M/F/U compared to sex estimate |
plan uses Gender |
Accept alias key columns; accept demographicSex/Gender/Sex for reported sex (M/F/U); use Sample_Group for plot colouring. |
| G60 | Methylation QC includes PCA control plot & principal component summary | not planned | Add optional PCA of control-probe metrics coloured by group to QC report (applies to genotyping too). |
| G61 | NextSeq550-scanned IDATs supported for select arrays | not mentioned | Test IDATs from NextSeq550 (format may differ in RunInfo/fields). |
| G62 | Genome builds 37 and 38 for PGx | plan PGx GRCh38 | Decide: support both or require 38 with explicit error. |
7.9 Documentation-level items
- G63: Provide a "product file compatibility matrix" (manifest revision ↔ EGT ↔ CN model ↔ cyto model ↔ DB version).
- G64: Provide end-to-end workflow recipes (genotyping, QC report, PGx, cyto, semi-custom) mirroring §1.15.
- G65: Document the recommended sample-QC gates for downstream steps (LogRDev ≤ 0.2, call rate ≥ 0.99, TGA ≥ 1.0).
8. External links referenced
- [ref-docs:llms.txt]
- [ref-docs:product-page]
- [ref-docs:downloads]
- [ref-docs:support-page]
- https://support.illumina.com/array/array_kits/infinium-global-diversity-pgx/product-files.html
- https://learn.microsoft.com/en-us/dotnet/core/install/linux-scripted-manual#deb-dependencies
- https://learn.microsoft.com/en-us/dotnet/core/install/linux-scripted-manual#rpm-dependencies
- https://www.illumina.com/techniques/microarrays/array-data-analysis-experimental-design/genomestudio.html
- https://www.illumina.com/Documents/products/technotes/technote_infinium_genotyping_data_analysis.pdf
- https://youtu.be/4JTrbMUbVN0?si=ZgRDLwN6umGBhv2G
- https://www.illumina.com/content/dam/illumina/gcs/assembled-assets/marketing-literature/custom-cluster-file-tech-note-m-gl-02142/custom-cluster-file-tech-note-m-gl-02142.pdf
- https://www.illumina.com/science/consortia/human-consortia.html
- https://knowledge.illumina.com/microarray/general/microarray-general-faq-list/000003871 (NextSeq550 arrays)
- http://my.illumina.com/
- https://knowledge.illumina.com/microarray/general/microarray-general-reference_material-list/000001531
- Internal: output-files.md (#snv_vcf_file, #genotype_call_file, #genotype_summary_files, #cnv_vcf_file, #cyto_vcf_file, #star_allele_dat, #using-infinium-i-nocall-information, #section-warningerror-messages-and-logs), qc-report.md, overview/our-features.md, release notes v1.0.0/v1.1.0 known issues.