# SimurgArray — Functional Specification > Companion to `SIMURGARRAY_MASTER_PLAN.md` (the "plan"). The plan defines algorithms, architecture, data sources > and the phased roadmap. **This document defines externally visible behaviour**: commands, options, validation, > file schemas, QC semantics, messages, edge cases and acceptance tests. Version 1.0 · 2026-09-27. > > It consolidates a documentation-level study of a commercial Infinium array-analysis product (the "reference > product"), whose public documentation was read in full. That study is archived under `_phase0_archive/site_analysis/` > as files A–E. About 200 suggested additions from their gap analyses were deduplicated and merged here. > No vendor binary was inspected (plan §2). Documented behaviour is emulated only where it helps users. > Documentation errors are listed in §12 and are **never** turned into requirements. --- ## 0. Conventions ### 0.1 Item format Every requirement has a stable ID, a trace tag and optional source tags: `- **CLI-021** \`§14 · P4\` **Title.** Requirement text … (src: A-G14, C-G1)` - **ID prefixes:** `SCOPE` scope, `CLI` command line, `IN` inputs, `OUT` genotyping outputs, `QC` control QC and the QC report, `CYTO` genome-wide CNV/LOH, `PCN` targeted PGx copy number, `PGX` star alleles and phenotypes, `MET` methylation control QC, `EDGE` edge cases and regression tests, `PERF` performance, `OQ` open questions. - **Trace tag:** the master-plan section(s), then the roadmap phase (P0–P10, or `v2` for the plan's v2 modules). The table in §13 is generated from these tags. - **Source tags** (`src:`) point at gap-analysis items in the archived study. `A-G5` means file A, item G5; `B-12` means file B, item 12; and so on for C, D and E. `plan` means the item restates or sharpens the plan. §13.2 checks that every gap item is covered. - **Keywords:** MUST, SHOULD and MAY as in RFC 2119. - **Behaviour markers:** - **[EMU]**: the reference product's documented behaviour, emulated on purpose. - **[DIFF]**: we deliberately do it differently. The reason is always given. - **[OURS]**: the reference documents nothing here, so this is SimurgArray's own definition. - **[INF]** and **[HYP]**: as in plan §0 (our inference; a hypothesis still to be tested). ### 0.2 Naming conventions (SimurgArray-wide) - CLI flags: lowercase kebab-case only. No mixed case and no case-sensitive look-alikes. - File names: lowercase snake_case. Per-sample files are named `..`, where `` is the *output stem* defined in IN-012. - TSV/CSV column names and JSON keys: snake_case. Reference-style column names are available only through the explicit compatibility options (OUT-066, QC-009). - Numbers are always formatted locale-independently: `.` as the decimal separator and no thousands separators (EDGE-001). - Timestamps in our outputs use ISO-8601 UTC (`2026-09-27T10:15:02Z`). Legacy formats are written only inside GTC fields, for interoperability (OUT-001). - Genomic coordinates are 1-based and fully closed in every TSV and JSON output. VCF follows the VCF 4.3 conventions exactly (OUT-041, CYTO-029). ### 0.3 Relationship to the plan Where this spec and the plan disagree about **I/O, naming, options or messages**, this spec wins. Where they disagree about **algorithms or numerics**, the plan wins. Each such conflict found during implementation MUST be logged in `docs/known_unknowns.md`, and both documents updated. --- ## 1. Scope & compatibility targets - **SCOPE-001** `§1 · P0` **Purpose.** SimurgArray reproduces the functional envelope of a modern Infinium secondary-analysis pipeline, locally and offline: - IDAT → GTC genotyping with control-probe QC and a sample summary; - GTC → SNV VCF and bedGraph; - a multi-dataset interactive QC report; - genome-wide CNV/LOH/mosaic calling with ISCN annotation (v2); - targeted PGx copy number and star-allele diplotypes with CPIC/DPWG phenotypes (v2); - methylation-array control QC (v2). (src: plan) - **SCOPE-002** `§1 · P0` **Emulation targets [EMU].** SimurgArray emulates these reference behaviours, because users and downstream tools depend on them: 1. **Workflow shape:** - genotype → VCF; - genotype (or control-only QC) → QC report; - genotype → VCF → targeted CN → star alleles → phenotype; - genotype → CNV/LOH → annotation. 2. **Content of the files that move between steps:** - control raw and QC metric tables; - the sample summary, with sheet columns passed through; - the SNV VCF (FORMAT `GT:GS:BAF:LRR`, merged multi-probe records); - the CNV VCF information content; - the PGx tables. 3. **Numeric defaults:** - GenCall cutoff 0.15; - QC 0.98 / 0.20; - per-type CNV size and probe minimums; - QUAL cap 60 and filters Q10 / Q7; - PGx batch 24/22 and training 96/90; - alternative solutions at ≥ 50 % of the top score; - low-GS review at 0.15. 4. **Control-probe metric formulas and QC status semantics** (two status axes, `null` disables a check). 5. **Sample-sheet semantics:** GenomeStudio-compatible sections, case-insensitive columns, name precedence, RootFolder. (src: A-G1, C-G1) - **SCOPE-003** `§1 · P0` **Deliberate differences [DIFF].** SimurgArray MUST differ from the reference product in each of the following. Each has its own item; this table is the index. | # | Reference behaviour | SimurgArray behaviour | Item | |---|---|---|---| | 1 | silently overwrites the output folder | refuses unless `--force`/`--resume` | CLI-010 | | 2 | duplicate IDAT pairs: one picked, OS-dependent | error `E_DUP_IDAT` (or a deterministic opt-in rule) | IN-022 | | 3 | CN probe thresholds < 10 silently reset | reset with a warning | CYTO-008 | | 4 | CNV VCF: `REF=N`, positive SVLEN for DEL, `GT=1/1` placeholder, ID/POS disagreements | VCF-4.3-correct padding-base POS, FASTA REF, signed SVLEN, meaningful GT, consistent IDs; `--compat` modes available | CYTO-029–CYTO-031, PCN-011 | | 5 | JSON numbers as strings, `"."` sentinels, stray key punctuation | typed JSON, `null`, JSON-Schema validated | OUT-075 | | 6 | output order and PGx scores differ by OS | byte-identical outputs across OS, CPU and thread count | OUT-076, PGX-029 | | 7 | licence server; results discarded when over quota | offline; computed results are never discarded | CLI-012, PGX-033 | | 8 | mixed-case, case-sensitive flags | lowercase kebab-case | CLI-002 | | 9 | unknown sex → silently male in CN calling | explicit policy + flag | CYTO-024 | | 10 | no ISCN for LOH/mosaic/≤ 1 kb; first event sometimes unannotated | ISCN for every reported event | CYTO-034 | | 11 | bedGraph smoothing corrupted LRR | raw tracks are never smoothed | OUT-052 | | 12 | own tabix writer without count metadata | htslib index with counts | OUT-038 | | 13 | cluster file names restricted to `[A-Za-z0-9_-]` | any name accepted | IN-045 | | 14 | BPM/CSV/EGT/IDAT mismatches silent or per-sample | pre-flight validation errors | IN-042–IN-044, IN-027 | | 15 | semi-custom arrays: PGx must run twice | single run; PGx uses whatever mapped content exists | PGX-030 | | 16 | per-step inconsistencies (raw vs effective size; GAINLOH counted in some places only) | one consistent rule, documented | CYTO-011, CYTO-033 | - **SCOPE-004** `§1.3 · P0` **Out of scope.** The following are not provided and MUST NOT be emulated: - cloud platform features: workgroups, credits, sharing, notifications, scanner-to-cloud upload, case creation on interpretation platforms; - licence servers or quota; - polyploid genotyping ("diploid only", as in the reference); - oncology CNV interpretation (`annotation_type` is always `constitutional`); - novel star-allele discovery. The local equivalents of two cloud features *are* in scope as optional v2 conveniences: the scan-registration gate (IN-026) and the planned-run watcher (CLI-037). (src: A-G13, D-26, D-27) - **SCOPE-005** `§1 · P0` **Species and builds.** - Human GRCh37 and GRCh38 for genotyping, VCF, QC and CNV/LOH. - PGx is GRCh38-only in v2.0; see PCN-015. - Non-human diploid organisms are supported for genotyping, GTC, QC and bedGraph, given a user-supplied FASTA and a *species profile* (IN-065). The bovine UMD3 build (BovineSNP50) is the non-human acceptance case. - A VCF for a non-human species needs a FASTA whose contig names match the manifest. (src: C-G22, A-G62) - **SCOPE-006** `§16 · P8` **Product test matrix.** Validation (plan §16.5) MUST include at least one dataset per manifest family: - genotyping: ASA, CGA, GCRA, GDA, GSA v3, GSA v4 (EX chemistry), JSA, PRS-type (GRCh37 only), BovineSNP50; - PGx: GDA-ePGx, GSAv4-ePGx, GCRA-ePGx (all GRCh38); - cyto: GDA-Cyto, GSA-Cyto, CytoSNP-850K (both iScan and NextSeq 550 IDATs); - methylation: EPIC v1, EPIC v2, MSA. Each is run on GRCh37 and GRCh38 where the product supports both. Families with no public data are recorded as gaps in the validation report. (src: C-G28) - **SCOPE-007** `§1 · P10` **Product-file compatibility matrix.** The user guide MUST publish a matrix linking manifest revision ↔ cluster file ↔ PGx CN model ↔ cyto model ↔ PGx DB ↔ annotation bundle. It lists, for each combination, which checks are errors and which are warnings (IN-049). `simurg inspect --compat ` prints the same matrix for the files the user supplies. (src: A-G63) - **SCOPE-008** `§3 · P10` **User runtime requirements.** - **Platforms:** Linux x86-64 and aarch64, macOS arm64. Windows is best-effort (via WSL) and not a release gate. - **Resources:** - Genotyping and QC: 4 cores and 8 GB RAM minimum; 8 cores and 32 GB recommended. - Disk: ≥ 30 GB free for GSA-size batches of 1 000 samples, plus the annotation bundle (CYTO-036). - PGx CN batches: memory scales with the batch, as documented in PCN-016. - **Operation:** - No elevated privileges needed; nothing is written outside the output, cache and tmp directories. - No network access at run time (offline-first). Downloads happen only in explicit `fetch`/`build` helper scripts. (src: A-G12, A-G13, A-G43) - **SCOPE-009** `§19 · P9` **Batch limits.** There is no hard sample cap. Every command MUST be tested at: - ≥ 1 152 samples for genotyping, cyto and methylation; - ≥ 384 samples for PGx; - 10 000 synthetic samples for genotyping and QC. Memory must stay flat in batch size wherever the computation is per-sample. (src: D-31, D-9) - **SCOPE-010** `§11 · v2` **Semi-custom and custom arrays.** - Genotyping, QC, VCF and CNV work on any Infinium genotyping manifest. - PGx uses every probe that maps to a PGx-DB variant, whichever part of the design it comes from. A second "commercial-files" run is never required [DIFF]. - Warnings are issued when a gene's commercial backbone probes are missing (PGX-030). (src: A-G42, D-20) --- ## 2. Command-line interface (`simurg`) ### 2.1 Structure and global behaviour - **CLI-001** `§14 · P1` **Command tree.** Plan §14's flat verbs are canonical. Grouped verbs are subcommands. ``` simurg ├── version # also: simurg --version ├── inspect ... [--compat] ├── ingest | normalize # plan §14 (developer/debug steps) ├── genotype # IDAT → GTC + controls + sample summary (P4) ├── gtc-to-vcf # GTC → .snv.vcf.gz (P6) ├── export bedgraph | plink | finalreport | locus-summary (P6) ├── qc │ ├── controls # IDAT + manifest → control tables, no cluster file (P5) │ ├── samples | loci | contamination | relatedness (P5) │ ├── report # multi-dataset HTML + table (P5) │ └── suggest-thresholds (P5) ├── cn # alias: cyto (v2) │ ├── build-model │ ├── call │ └── annotate ├── pgx (v2) │ ├── copy-number call | train │ ├── star-allele call │ ├── phenotype # alias: annotate │ ├── coverage │ └── db build ├── methyl (v2) │ ├── qc │ └── convert-manifest ├── train # ArrayTrain (plan §12) (v2) ├── pipeline run | plan | watch (P6; watch v2) ├── compare (P7) └── synth (P1+) ``` Every group and leaf accepts `--help`. `simurg ` with no leaf prints the group help and exits with 2. (src: A-G1, A-G44, plan) - **CLI-002** `§14 · P1` **Syntax rules.** - Options are lowercase kebab-case. Boolean options come in `--x/--no-x` pairs where a default-on switch exists. - List-valued options accept either a comma list with no spaces (`a,b`; surrounding spaces are trimmed with a warning) or a repeated flag. - Paths may contain spaces and non-ASCII characters. - Unknown options are errors (exit 2) and come with a "did you mean" suggestion. (src: A-G11) - **CLI-003** `§14 · P10` **Migration aliases.** A documented table maps reference-style verbs to simurg verbs: - `genotype call` → `genotype` - `genotype gtc-to-vcf` → `gtc-to-vcf` - `genotype gtc-to-bedgraph` → `export bedgraph` - `qc call` → `qc controls` - `cyto call|annotate` → `cn call|annotate` - `pgx copy-number call|train` → identical - `pgx star-allele call` → identical - `pgx star-allele annotate` → `pgx phenotype` The two-word forms are accepted as hidden aliases, with an informational note. Mixed-case reference flags (for example the threshold flags) are *not* accepted; the error message names the simurg equivalent. (src: A-G1) - **CLI-004** `§14 · P1` **Global options** (accepted by every leaf command). | Option | Type | Default | Validation / notes | |---|---|---|---| | `--threads N` | int | `0` = auto: physical cores available to the process, respecting CPU affinity and cgroup quota | 0–1024; results MUST NOT depend on N (plan §15) | | `--log-level` | enum `debug\|info\|warning\|error` | `info` | | | `--debug` | flag | off | sets `debug` level **and** includes tracebacks in error records | | `--log-json` (alias `--json-log`) | flag | off | JSON Lines on stderr, schema in CLI-007 | | `--log-file PATH` | path | `/logs/-.log` | always JSON Lines, regardless of `--log-json` | | `--quiet` / `--no-progress` | flag | progress bars only when stderr is a TTY | | | `--seed N` | int | `42` | recorded in the run manifest | | `--config FILE` | YAML | none | see CLI-019 | | `--force` | flag | off | allows replacing existing outputs (CLI-010) | | `--resume` | flag | off | reuses completed per-sample outputs whose input hashes match (plan §14 content-addressed cache) | | `--tmp-dir DIR` | dir | `/.simurg_tmp` | removed on success | | `--help`, `--version` | flag | | `--version` prints `SimurgArray +` | (src: A-G2, A-G3, A-G7, plan) - **CLI-005** `§14, §15 · P1` **Option precedence.** 1. Explicit CLI option. 2. The command's section in the `--config` YAML. 3. The named profile (QC-024), if any. 4. The built-in default. The only environment variables read are `SIMURG_CACHE_DIR` (IN-064) and `NO_COLOR`. The *effective* value of every option and threshold, with its source (`cli|yaml:|profile:|default`), MUST be recorded: - in `run_manifest.json`; - in VCF headers (`##simurg_options=`); - in JSON outputs (a `parameters` object); - in the QC table's Thresholds sheet (QC-045). (src: A-G30, E-47) - **CLI-006** `§14 · P1` **Input-set rule.** Commands that take a sample set accept a folder (`--idats DIR` or `--gtcs DIR`), a sheet (`--sheet FILE`), or both. - **Sheet only:** the folder is the sheet's `RootFolder` (IN-008). If it has none → `E_NO_INPUT_FOLDER`. - **Folder only:** every discovered sample is processed. - **Both:** the sheet selects and annotates samples; the CLI folder overrides `RootFolder`, with an info message if they differ. - **Neither:** `E_NO_INPUT` (exit 2). (src: A-G5, plan) - **CLI-007** `§14 · P1` **Logging.** - Human-readable logs go to stderr. With `--log-json`, each record is one JSON object per line: ```json {"schema":"simurg.log/1","ts":"2026-09-27T10:15:02.123Z","level":"warning", "code":"W_MERGE_AMBIGUOUS","msg":"Failed to combine genotypes (ambiguous) - rs1 (InfiniumII): AA, rs1_dup (InfiniumI): TT", "command":"gtc-to-vcf","run_id":"01J9…","step":"merge","sample_id":"204753010023_R01C01", "locus":"rs1","file":null,"count":1,"elapsed_ms":8123,"exc":null} ``` - `exc` holds the formatted traceback only with `--debug`. - `code` is a stable catalogue ID (CLI-016); `msg` is free text that may change between versions. - Unknown fields MUST be ignored by consumers. (src: A-G2, A-G3) - **CLI-008** `§14 · P1` **Warning rate limiting.** - Each warning code is printed at most 20 times per sample, then once as a summary: `W_… repeated N more times`. - Every occurrence is still written to the audit tables (plan §8.2), and the run summary counts them per code. (src: B-5) - **CLI-009** `§14 · P1` **Run summary.** Each command ends with a summary: - samples succeeded / failed / skipped; - counts per warning code; - output paths; - wall time. It goes to stderr and into `run_manifest.json` (`summary` object). (src: E-18, plan) - **CLI-010** `§14 · P1` **Overwrite policy [DIFF].** - If any file the command would write already exists, the command stops before doing any work: `E_OUTPUT_EXISTS` lists up to 10 paths, exit 2. - `--force` replaces them. - `--resume` keeps outputs whose recorded input hashes and parameters match, and recomputes the rest. - Files not produced by the current command are never deleted. - Exception: `qc controls` MAY add its control tables to an existing genotype output folder that lacks them (QC-014). (src: A-G4, E-6) - **CLI-011** `§14 · P1` **Atomic writes.** Every output is written to `.part` in the same directory, then fsynced and renamed. An interrupted run MUST NOT leave a truncated file under its final name. (src: plan §14) - **CLI-012** `§14 · P1` **Never discard results [DIFF].** Per-sample outputs are finalized as soon as each sample completes. A later failure (another sample, batch step or report) does not remove them. Batch files (summary, QC tables) are rewritten at the end, and are also written if the run fails part-way. (src: A-G10) - **CLI-013** `§14, §15 · P1` **Threading.** - Parallelism is per sample (a process pool) plus, inside a sample, vectorized/Rust threads (P9). The total never exceeds `--threads`. - bgzip compression uses htslib worker threads taken from the same budget. - Changing `--threads` MUST NOT change any output byte (EDGE-022). (src: A-G7) - **CLI-014** `§14 · P1` **Exit codes.** | Code | Meaning | |---|---| | 0 | success; warnings are allowed | | 1 | internal error (a bug). The message asks for a `--debug` rerun | | 2 | usage or pre-flight validation error; no per-sample work started. Examples: bad option, missing file, schema or mismatch errors, output exists, batch minimum not met | | 3 | partial failure: ≥ 1 sample failed and ≥ 1 succeeded. Outputs for the successful samples are complete, and the failed samples are listed in the summary | | 4 | the run completed but no sample succeeded (extension of plan §14) | | 130 / 143 | interrupted by SIGINT / SIGTERM (CLI-015) | (src: A-G10, plan) - **CLI-015** `§14 · P1` **Interruption.** On SIGINT/SIGTERM: 1. Stop scheduling new work. 2. Let in-flight atomic writes finish or abort them; either way no partial final files. 3. Write `run_manifest.json` with `status: "interrupted"`. 4. Exit with 130/143. A later `--resume` continues the run. (src: plan) - **CLI-016** `§14 · P1` **Message catalogue.** Every error and warning has a stable code. The catalogue (`docs/messages.md`, generated from `simurg/core/errors.py`) is part of the acceptance tests (EDGE-*). Minimum set: | Code | Level | Trigger | |---|---|---| | `E_NO_INPUT`, `E_NO_INPUT_FOLDER` | error | CLI-006 | | `E_OUTPUT_EXISTS` | error | CLI-010 | | `E_FILE_NOT_FOUND`, `E_FILE_UNREADABLE` | error | any missing or unreadable input, e.g. a manifest path | | `E_SHEET_*` (`DUP_COLUMN`, `RESERVED`, `TOO_LONG`, `BARCODE_FORMAT`, `NAME_EMPTY`, `NAME_COLLISION`, `DUP_SAMPLE`, `NO_KEY_COLUMNS`, `EMPTY`, `DELIMITER`) | error | IN-001 – IN-016 | | `W_SHEET_EMPTY_COLUMN`, `W_SHEET_BARCODE_REPAIRED`, `W_SHEET_ROOTFOLDER_OVERRIDDEN` | warning | IN-004, IN-007, IN-008 | | `E_DUP_IDAT`, `W_IDAT_UNPAIRED`, `E_IDAT_FORMAT`, `E_IDAT_ENCRYPTED`, `W_IDAT_NAME_MISMATCH` | error / warning | IN-020 – IN-023 | | `E_IDAT_MANIFEST_MISMATCH`, `W_IDAT_LOW_ADDRESS_COVERAGE` | error / warning | IN-027 | | `E_GTC_CORRUPT` (sample skipped), `W_GTC_FOREIGN`, `E_GTC_UNSUITABLE` | warning / error | IN-031 – IN-033 | | `E_BPM_CSV_MISMATCH`, `E_EGT_MANIFEST_MISMATCH`, `E_EGT_LOAD`, `W_EGT_NAME` | error / warning | IN-042 – IN-045 | | `E_FAI_MISMATCH`, `E_CONTIG_MISMATCH`, `E_BUILD_MISMATCH`, `E_PGX_ALT_CONTIG_MISSING` | error | IN-060 – IN-063 | | `W_REF_NOT_QUERIED`, `W_SKIP_UNMAPPED`, `W_SKIP_INTENSITY_ONLY`, `W_SKIP_INDEL_CONTEXT`, `W_SKIP_INDEL_REF`, `W_INDEL_PARTIAL_MATCH`, `W_SKIP_EXCLUDED` | warning | OUT-019 | | `W_MERGE_INCONSISTENT`, `W_MERGE_AMBIGUOUS`, `W_MERGE_ALL_NC`, `W_INF1_NC_CONFLICT` | warning | OUT-031 | | `E_AUX_INVALID` | error | OUT-035 | | `W_CTRL_MISSING`, `W_CTRL_ZERO_DIVISOR` | warning | QC-007, QC-008 | | `E_QC_*` (`NO_DATASETS`, `DATASET_INCOMPLETE`, `LABEL_COUNT`, `LABEL_DUPLICATE`, `MIXED_TYPES`, `FORMAT`, `CONFIG_MISSING`, `CONFIG_EXT`, `CONFIG_PARSE`, `CONFIG_STRUCTURE`, `THRESHOLD_KEY`, `THRESHOLD_VALUE`, `THRESHOLD_RANGE`) and `W_QC_LABEL_SANITIZED`, `W_QC_UNMATCHED_SAMPLES` | error / warning | QC-020 – QC-043 | | `E_MODEL_MANIFEST_MISMATCH`, `E_MODEL_CLUSTER_MISMATCH` / `W_MODEL_CLUSTER_MISMATCH`, `E_MODEL_VERSION` | error / warning | IN-050, IN-051 | | `E_PGX_BATCH_TOO_SMALL`, `W_PGX_BATCH_BELOW_RECOMMENDED`, `E_PGX_TRAIN_TOO_SMALL` | error / warning | PCN-004, PCN-005 | | `E_PGX_VCF_UNSUITABLE`, `E_PGX_VCF_UNINDEXED`, `E_PGX_CN_MISSING` | error | PGX-011 | | `W_CN_PROBE_FLOOR`, `W_CN_THRESHOLD_EXCEEDS_GENOME`, `I_CN_SEX_RESOLVED`, `W_CN_SEX_UNRESOLVED`, `E_CN_SIDECAR_MISSING`, `E_CN_VCF_UNINDEXED` | mixed | CYTO-* | | `E_METH_*` and `W_METH_*` | mixed | MET-015 | (src: B-33, C-G21, A-G56, B-5) - **CLI-017** `§14 · P1` **Paths.** - Relative paths on the CLI resolve against the CWD. Relative paths inside a sheet or config resolve against that file's directory. - Symlinks are followed, with loop detection. - Output paths are printed as absolute paths. (src: E-4) - **CLI-018** `§13 · P1` **Output layout.** - Default `--layout nested` (plan §13): per-sample files go under `/samples//`, batch files under `/`. - `--layout flat` writes everything into `/`, as the reference does. - Every downstream command discovers inputs recursively, so both layouts work interchangeably. This includes the star-allele step, which needs SNV and CN VCFs "in one folder" in the reference. (src: A-G44, plan) - **CLI-019** `§14 · P1` **Configuration file.** - `--config` takes YAML with an optional `version: 1`. It has one top-level section per command (`genotype:`, `gtc_to_vcf:`, `qc:`, `cn:`, `pgx:`, `methyl:`) plus `runtime:`. - Keys are the snake_case option names. Unknown keys → `E_CONFIG_KEY` (exit 2). - `pipeline run` uses the same file, extended with `inputs:`, `reference:` and `outputs:` (plan §14 example). In `outputs:`, `gtc`, `vcf`, `bedgraph`, `plink`, `final_report`, `locus_summary`, `qc_report`: true/false. - `pipeline plan` warns when `final_report: true` and loci × samples > 1e8, and suggests turning it off (about 400 GSA-size samples ≈ 50 GB). (src: D-30, plan) ### 2.2 Commands - **CLI-020** `§14 · P1` **`simurg inspect ...`** - Prints the header/TOC of IDAT, BPM, CSV manifest, EGT, GTC, sheet, VCF, CN model, PGx DB or annotation bundle. - `--json` gives machine-readable output. - `--compat` prints the compatibility matrix for the given files (SCOPE-007). (src: plan) - **CLI-021** `§14 · P4` **`simurg genotype`** (IDAT → GTC + control tables + sample summary). | Option | Type | Default | Validation | |---|---|---|---| | `--idats DIR` | dir | – | CLI-006; recursive discovery, IN-020 | | `--sheet FILE` | CSV | – | IN-001 … | | `--bpm FILE` | BPM | **required** | IN-040 | | `--egt FILE` | EGT | **required** | IN-043/IN-044 | | `--csv FILE` | CSV manifest | optional | if given, checked against the BPM (IN-042) and used for control names (QC-001) | | `--out DIR` | dir | **required** | CLI-010 | | `--cutoff X` | float | `0.15` | 0 ≤ X ≤ 1; GenCall score < X → no-call | | `--norm-version` | enum `1.2.0\|1.1.2` | `1.2.0` | plan §7.2 | | `--exact/--clean` | flag | `--exact` | plan §0 | | `--gtc/--no-gtc` | flag | on | off keeps calls only in the Parquet output | | `--controls/--no-controls` | flag | on | writes `controls_raw.tsv`, `controls_qc.tsv` (QC-009, QC-011) | | `--summary-format` | enum `tsv\|csv\|both` | `tsv` | plus `.json` always | | `--compat-column-names` | flag | off | OUT-066 | | `--on-duplicate-idat` | enum `error\|first\|newest` | `error` | IN-022 | | `--assay-control-probes` | list | `TGA_Ctrl_5716` | OUT-062 | | `--species-profile` | name/file | `human` | IN-065 | Outputs: GTC (OUT-001), `sample_summary.{tsv,json}` (OUT-060), control tables (QC-009/011), `run_manifest.json`. (src: A-G14, A-G7, plan) - **CLI-022** `§14, §8 · P6` **`simurg gtc-to-vcf`.** | Option | Type | Default | Validation / notes | |---|---|---|---| | `--gtcs DIR` / `--sheet FILE` | dir / CSV | – | CLI-006 | | `--bpm FILE` | BPM | **required** | | | `--csv FILE` | CSV manifest | **required** for indels | must contain `SourceSeq`. If absent, indels are skipped with `W_SKIP_INDEL_CONTEXT` and a one-time error-level notice | | `--fasta FILE` | FASTA | **required** | IN-060 | | `--out DIR` | dir | **required** | | | `--no-bgzip` | flag | off | OUT-010 | | `--genome-cache/--no-genome-cache` | flag | cache on | IN-064 | | `--auxiliary-loci FILE` | VCF | – | OUT-035 | | `--exclude-loci FILE` (alias `--filter-loci`) / `--include-loci FILE` | text | – | OUT-036 | | `--unsquash-duplicates` | flag | off | OUT-034 | | `--inf1-nocall-informative` | flag | off | OUT-037 | | `--merge` | flag | off | multi-sample BCF (OUT-040) | | `--drop-ambiguous` | flag | off | plan §8.2 | | `--info-fields` | list | none | OUT-012 | | `--extra-format` | list `X,Y,NORMX,NORMY,THETA,R` | none | plan §8.3 | | `--filter-gs X` | float | off | adds FILTER `LowGS` when GS < X | | `--contig-alias` | enum `strict\|auto\|FILE` | `strict` | IN-061 | | `--ploidy-policy` | enum `sex-aware\|diploid` | `sex-aware` | OUT-018 | | `--merge-snv-indel` | flag | off | OUT-021 | (src: A-G18, A-G19, A-G20, A-G9, A-G8, plan) - **CLI-023** `§14 · P6` **`simurg export …`** - `export bedgraph --gtcs/--sheet --bpm [--no-gzip]`: OUT-050. - `export plink --vcfs|--gtcs … [--multiallelic split|drop]`: OUT-043. - `export finalreport --gtcs --bpm --csv [--matrix]`: OUT-071. - `export locus-summary --gtcs --bpm --egt`: OUT-072. None of them re-genotypes: they read GTCs plus the manifest. (src: A-G26) - **CLI-024** `§14, §9 · P5` **`simurg qc controls`** (control-only QC, no cluster file). | Option | Type | Default | Validation | |---|---|---|---| | `--idats DIR` / `--sheet FILE` | | | CLI-006 | | `--csv FILE` or `--bpm FILE` | manifest | one required | control definitions (QC-001) | | `--array-type` | enum `genotyping\|methylation` | **required** | selects the metric set (QC-005/QC-006) | | `--out DIR` | dir | **required** | QC-014 exception to CLI-010 | | `--bg-offset X` | float | `3000` | ≥ 0 | (src: A-G27, C-G2) - **CLI-025** `§14, §9 · P5` **`simurg qc samples | loci | contamination | relatedness`.** These follow plan §9.2–§9.4 and §7.9 and write `sample_qc.tsv`, `locus_summary.tsv`, `contamination.tsv` and `relatedness.tsv`. They take `--gtcs`/`--sheet`, the manifest and `--out`. `relatedness` also takes `--pedigree-from-sheet` (plan §9.4). (src: plan) - **CLI-026** `§14, §9.5 · P5` **`simurg qc report`.** | Option | Type | Default | Validation | |---|---|---|---| | `--data` | list of dirs | **required** | one folder, a comma list, or one parent folder (QC-041) | | `--label` | list | folder names | count must equal the number of datasets (QC-042) | | `--config FILE` | YAML | none | QC-020/QC-021 | | `--profile NAME` | enum (QC-024) | none | | | `--threshold-call-rate X` | float | `0.98` | 0 ≤ X ≤ 1 | | `--threshold-lrr-sd X` | float | `0.20` | X ≥ 0 | | `--threshold KEY=VALUE` | repeatable | – | any QC-020 key; `VALUE` may be `null` | | `--output-format` | enum `xlsx\|csv\|both` | `xlsx` | anything else → `E_QC_FORMAT` | | `--report-name STEM` | str | `qc_report_YYYY_MM_DD` / `qc_table_YYYY_MM_DD` | QC-044 | | `--drop-metadata COLS` / `--keep-metadata COLS` / `--no-metadata` | list / flag | keep all | QC-046 | | `--split-by` | enum `none\|dataset\|plate\|run` | `none` | QC-057 | | `--out DIR` | dir | CWD | | (src: A-G28, A-G29, A-G30, A-G31, C-G6, C-G17, C-G20) - **CLI-027** `§9 · P5` **`simurg qc suggest-thresholds --data … [--k 4] [--min-samples 30] --out FILE.yaml`**: QC-026. (src: C-G8) - **CLI-028** `§10 · v2` **`simurg cn call`** (alias `cyto call`). | Option | Type | Default | Range / validation | |---|---|---|---| | `--gtcs DIR` / `--sheet FILE` | | | CLI-006; GTCs must qualify (IN-031) | | `--model FILE` | cyto model | **required** | IN-051 | | `--summary FILE` | sample_summary.tsv | auto-discovered next to the GTCs | source of sex and LRR SD | | `--fasta FILE` | FASTA | optional; recommended | REF padding base (CYTO-029) | | `--out DIR` | dir | **required** | | | `--min-del-probes` / `--min-del-size` | int / kb | 10 / 25 | probes 0–250000 (floored to 10, CYTO-008); size 0–250000 kb | | `--min-dup-probes` / `--min-dup-size` | int / kb | 10 / 50 | same | | `--min-loh-probes` / `--min-loh-size` | int / kb | 500 / 3000 | same | | `--min-gainloh-probes` / `--min-gainloh-size` | int / kb | 300 / 500 | same | | `--min-mdel-probes` / `--min-mdel-size` | int / kb | 10 / 25 | same | | `--min-mdup-probes` / `--min-mdup-size` | int / kb | 10 / 50 | same | | `--min-mloh-probes` / `--min-mloh-size` | int / kb | 500 / 3000 | same | | `--smoothing K` | int | 5 | 0–1000 probes each side (CYTO-004) | | `--max-mosaic-fraction F` | float | 1.0 | 0 < F ≤ 1 (CYTO-020) | | `--min-mosaic-fraction F` | float | 0.10 | 0 ≤ F < 1 (`LowFraction`, CYTO-021) | | `--filter-qual Q` | int | 10 | 0–60 (FILTER `Q10`) | | `--cn-cap N` | int | 4 | 3–8 | | `--gc-correct/--no-gc-correct` | flag | on | | | `--unknown-sex-policy` | enum `skip-sex-chromosomes\|female\|male\|error` | `skip-sex-chromosomes` | CYTO-024 | | `--gt-mode` | enum `vcf\|compat` | `vcf` | CYTO-031 | | `--svlen-mode` | enum `signed\|positive` | `signed` | CYTO-029 | | `--mask-bed FILE` | BED | built-in centromere/gap mask | CYTO-017 | | `--no-bgzip` | flag | off | warns that `cn annotate` needs bgzip + index (CLI-039) | (src: A-G45, A-G47, A-G48, D-2, D-3, C-G24) - **CLI-029** `§10 · v2` **`simurg cn annotate`** (alias `cyto annotate`). - Required: `--vcfs DIR` (the `cn call` output) and `--annotation-bundle FILE` (CYTO-036). - Optional: - the same 14 `--min-*` options, defaulting to the values recorded by `cn call`; they can only tighten, never loosen; - `--min-qual Q` (default 20, 0–60); - `--size-basis effective|raw` (default `effective`, CYTO-011); - `--iscn-min-size bp` (default 0); - `--out DIR`. (src: A-G49, A-G53, A-G54, D-2) - **CLI-030** `§10 · v2` **`simurg cn build-model --bpm --fasta [--csv] --out model.simurg-cyto`**: CYTO-038. (src: A-G52) - **CLI-031** `§11 · v2` **`simurg pgx copy-number call`.** | Option | Type | Default | Validation | |---|---|---|---| | `--gtcs DIR` / `--sheet FILE` | | | GTCs must qualify (IN-031) | | `--model FILE` | PGx CN model | **required** | IN-050 | | `--fasta FILE` | GRCh38 FASTA with alt contigs | **required** | IN-062 | | `--out DIR` | dir | **required** | recommended: the SNV VCF folder (PCN-014) | | `--min-batch N` | int | 24 | ≥ 1 | | `--min-passing N` | int | 22 | ≤ `--min-batch` | | `--recommended-passing N` | int | 96 | warning only | | `--lrr-sd-max X` | float | 0.20 | passing means LRR SD **<** X (PCN-004) | | `--cluster-mismatch` | enum `error\|warn` | `error` | IN-050 | | `--no-bgzip` | flag | off | | (src: A-G33, D-11, E-39) - **CLI-032** `§11 · v2` **`simurg pgx copy-number train`.** | Option | Type | Default | Validation | |---|---|---|---| | `--gtcs` / `--sheet` | | | | | `--bpm FILE` | BPM | **required** | | | `--csv FILE` | CSV manifest | optional | source of `Assay Format` | | `--fasta FILE` | FASTA | **required** | | | `--mask FILE` | TSV | `.mask.tsv` next to the BPM if present | IN-052 | | `--platform` | enum `auto\|LCG\|EX` | `auto` = from the CSV `Assay Format` heading | error if `auto` and no CSV | | `--min-samples N` / `--min-passing N` | int | 96 / 90 | | | `--max-samples N` | int | 3000 | seeded random subset above this, with a note | | `--train-lrr-sd-max`, `--train-call-rate-min`, `--train-assay-control-min` | float | 0.20 / 0.99 / 1.0 | exclusion rules (PCN-005) | | `--genome-cache/--no-genome-cache` | flag | on | | | `--out DIR` | dir | CWD | | (src: A-G34, A-G35, A-G36) - **CLI-033** `§11 · v2` **`simurg pgx star-allele call`.** - Required: `--vcfs DIR[,DIR]` (SNV + PGx CN VCFs; recursive; symlinks allowed), `--db FILE` and `--out DIR`. - Optional: - `--population NAME` (prior; default `global`); - `--max-candidates N` (10); - `--feasible-ratio R` (0.5); - `--low-gs X` (0.15); - `--no-cn` (run without CN VCFs; CN-dependent genes are reported as `not_assessed`); - `--allow-foreign-vcf`. - Writes `.pgx_calls.parquet` plus preliminary TSVs. (src: A-G39, A-G41, B-28, E-40) - **CLI-034** `§11 · v2` **`simurg pgx phenotype`** (alias `annotate`). - Required: `--calls DIR`. - Optional: `--guideline cpic|dpwg` (default `cpic`) and `--out DIR`. - Writes the TSV set (PGX-026) and `.pgx.json`. It can be re-run with another guideline without re-calling. (src: A-G41, B-28, D-16) - **CLI-035** `§11 · v2` **`simurg pgx db build --sources DIR --manifests BPM[,…] --out FILE`**: PGX-002. **`simurg pgx coverage --bpm --db`**: PGX-035. (src: A-G40) - **CLI-036** `§1.2 · v2` **`simurg methyl qc`.** - Required: `--idats`/`--sheet`, `--manifest CSV` and `--out`. - Optional: - `--profile epic|msa|custom`; - `--threshold KEY=VALUE`, `--config`; - `--pval-threshold` (0.05); - `--bg-offset` (3000); - `--beta-offset` (0); - `--min-prop-passing` (off); - `--pca-probes` (10000); - `--pca-max-samples-full` (100); - `--ffpe` (sets restoration to 1). - **`simurg methyl convert-manifest`**: MET-003. (src: C-G26, D-21, D-22) - **CLI-037** `§14 · P6 (watch: v2)` **`simurg pipeline run|plan|watch config.yaml`.** - `plan` = dry run: it resolves inputs, validates everything (IN-049) and prints the step DAG, output sizes and warnings. - `run` executes, with `--resume` support. - `watch` (planned-run mode): polls `inputs.idat_dir` until every sheet sample has an IDAT pair; with `--scan-metrics`, a sample whose scan registration fails counts as failed (IN-026). - It **aborts** if failed samples > `max_failed_scans` (default 0). - Otherwise it runs with the passing samples. - It writes `sample_status.tsv` (`sample_id, status ∈ {ready, failed, pending}, reason`). (src: D-27, D-30) - **CLI-038** `§14 · P7, P1` **`compare`, `synth`, `train`** follow plan §14, §16.4 and §12. (src: plan) - **CLI-039** `§14 · P6` **Cross-step guards.** - Using `--unsquash-duplicates` or an exclude/include list in `gtc-to-vcf` records this in the VCF header; `pgx star-allele call` refuses such VCFs (PGX-011). - `pipeline plan` also warns *before* running when the config combines these options with a PGx step. - `cn call --no-bgzip` warns if the configured pipeline contains `cn annotate`. (src: A 1.14, A-G39, A-G9) - **CLI-040** `§14 · P10` **Workflow recipes** in the user guide, each as copy-pasteable commands plus a `pipeline` YAML: 1. genotyping (IDAT → GTC → VCF); 2. QC report (genotype or control-only, then the report; multi-run trends); 3. PGx (genotype → VCF → CN → star alleles → phenotype; optional CN training); 4. cytogenetics (genotype → cn call → cn annotate → bedGraph); 5. semi-custom arrays (single run, SCOPE-010); 6. methylation QC. Each recipe states the recommended sample gates (QC-035). (src: A-G64, A-G65) --- ## 3. Inputs ### 3.1 Sample sheet - **IN-001** `§6.6 · P1` **Format.** - The sheet is a CSV file: UTF-8 (a BOM is tolerated), comma-delimited, RFC 4180 quoting, LF or CRLF line endings. - Optional sections: `[Header]` (key,value rows), `[Manifests]` (**ignored**) and `[Data]` (the table). - A sheet with no section markers is valid: its first non-empty line is the header row. - Section names are matched case-insensitively. - The file is rejected with `E_SHEET_DELIMITER` and a hint in two cases: - semicolon- or tab-delimited files (e.g. from European Excel); - JSON sheets. - An empty `[Data]` section, or a sheet with no rows, → `E_SHEET_EMPTY`. (src: A-G58, E-8, B-31) - **IN-002** `§6.6 · P1` **Key columns.** - Required: `SentrixBarcode_A` and `SentrixPosition_A`. - Also accepted, as aliases for the same pair: `beadChipName` and `sampleSectionName`. - Missing both forms → `E_SHEET_NO_KEY_COLUMNS`. - Optional recognised columns: - `Sample_ID`, `Sample_Name`, `Sample_Plate`, `Sample_Well`, `Sample_Group`; - reported sex: `Gender`, `Sex` or `demographicSex`; - `Parent1`, `Parent2`, `Replicate`. (src: A-G59, E-8, plan §6.6) - **IN-003** `§6.6 · P1` **Case-insensitive columns.** - Header names are matched after Unicode case-folding and trimming. - Two columns that become equal after folding are an error: `E_SHEET_DUP_COLUMN`, with the message "Duplicate column '' (column names are case-insensitive). Rename or remove one of: , ". (src: A-G56, E-9) - **IN-004** `§6.6 · P1` **Empty header cells.** Columns whose header is empty are ignored, with one `W_SHEET_EMPTY_COLUMN` warning. Examples: `SentrixBarcode_A,SentrixPosition_A,,` and a single trailing comma. Such columns MUST NOT count as duplicates of each other, and MUST NOT produce empty-key fields in any output. (src: E-9) - **IN-005** `§6.6 · P1` **Reserved column names.** A sheet column must not collide (case-insensitively) with any column that SimurgArray itself writes into the sample summary. That covers: - every OUT-060 column name; - their reference-style spellings under OUT-066, such as `Sex Estimate`, `Call Rate`, `Log R Ratio Std Dev` and `Sample Folder`. A collision → `E_SHEET_RESERVED`. With `--namespace-sheet-columns`, all pass-through columns are prefixed `sheet_` instead. (src: A-G56, E-9) - **IN-006** `§6.6 · P1` **Length limit.** Column names and cell values are limited to 500 characters (Unicode code points). A longer one → `E_SHEET_TOO_LONG`, naming the row, the column and its length. (src: A-G56, E-9) - **IN-007** `§6.6 · P1` **Barcode and position validation.** - **Barcode:** after trimming it must match `^[0-9]{10,14}$`. - Values like `204753010023.0` are repaired to `204753010023`, with `W_SHEET_BARCODE_REPAIRED`. - Scientific notation (`2.04753E+11`, `2,04753E+11`) cannot be repaired, because precision is lost. It is rejected with `E_SHEET_BARCODE_FORMAT` and the hint "format the SentrixBarcode_A column as a number with 0 decimals (or as text) before saving". - **Position:** must match `^[Rr][0-9]{2}[Cc][0-9]{2}$` and is normalised to upper case. (src: A-G56, E-9) - **IN-008** `§6.6 · P1` **RootFolder.** - `[Header] RootFolder,` names the IDAT (or GTC) root. A relative path resolves against the sheet's directory. - A CLI folder overrides it (`W_SHEET_ROOTFOLDER_OVERRIDDEN`, logged at info level when the two paths are equal). - Other `[Header]` keys are kept verbatim in `run_manifest.json` under `sheet_header`. (src: A-G57, E-8) - **IN-009** `§6.6 · P1` **Sample identity.** `sample_id` is always `_`. A duplicate `sample_id` among sheet rows → `E_SHEET_DUP_SAMPLE`. `--on-duplicate-sample drop` MAY instead keep the first row and warn, as the methylation pipeline of the reference does. (src: B-31, D-24) - **IN-010** `§6.6 · P1` **Sample-name precedence [EMU].** 1. The sheet's `Sample_Name`, if the column exists. 2. The sample name stored in the IDAT (field 406), or in the GTC (field 10). 3. `sample_id`. Rules for `Sample_Name` when present: - It must be non-empty in **every** row (`E_SHEET_NAME_EMPTY`). - It must not equal a *different* sample's `sample_id` (`E_SHEET_NAME_COLLISION`). - It must be unique across the sheet (`E_SHEET_NAME_COLLISION`) [OURS: required so that output file names are unique]. (src: A-G16, E-10) - **IN-011** `§6.6 · P1` **Per-command resolution.** - Names are resolved independently by each command, and the sheet given to that command is the source of truth. - The name inside a GTC is **never** rewritten: `genotype` stores the IDAT name, or empty. - A later command given a sheet uses the sheet's names in its outputs: VCF sample column, summaries, JSON. (src: A-G16, E-10) - **IN-012** `§13 · P1` **Output stem.** `` in file names is the resolved sample name after sanitisation: - characters outside `[A-Za-z0-9._-]` become `_`; - leading dots are stripped; - length is capped at 200. If two samples sanitise to the same stem → `E_SHEET_NAME_COLLISION`. The unsanitised name is kept inside the files. (src: plan §13) - **IN-013** `§6.6 · P1` **Metadata pass-through.** - Every sheet column that is not a key column is copied **verbatim** into `sample_summary.{tsv,json}`: the header as written in the sheet, the values as strings, sheet column order preserved. - They are placed after the fixed columns (OUT-060). - The QC report and QC table receive them from the summary (QC-040). (src: A-G15, E-11, B-9) - **IN-014** `§6.6 · P1` **Reported sex.** - Values from `Gender`, `Sex` or `demographicSex` are normalised: `M`/`Male`/`1` → `M`, `F`/`Female`/`2` → `F`, everything else (including empty) → `U`. - If more than one of these columns is present and they disagree → warning, and the first in the order `Sex`, `Gender`, `demographicSex` wins. - The result goes into `sex_reported`, and is compared with `sex_est` in `sex_check` (OUT-060). (src: A-G59) - **IN-015** `§6.6 · P1` **Plate, well and group.** - `Sample_Plate` and `Sample_Well` from the sheet override the IDAT fields 408/409. They drive plate heatmaps (QC-052) and the plate/well columns of `controls_qc.tsv`. - `Sample_Group` is the default colour field in plots: control PCA (QC-060) and methylation PCA (MET-013). (src: C-G3, A-G59, A-G60) - **IN-016** `§6.6 · P1` **Sheet vs files.** - A sheet row with no matching IDAT/GTC gets a summary row with status `MISSING_INPUT` and counts toward exit 3/4. - Files not listed in the sheet are ignored; an info message gives their count. (src: plan) - **IN-017** `§6.6, §19 · P1` **Scale.** Parsing and matching are O(n) in rows plus files. A 10 000-row sheet against 10 000 IDAT pairs must parse and match in < 2 s (PERF-007). (src: E-7) - **IN-018** `§6.6 · P5` **Parent/replicate columns.** `Parent1`, `Parent2` and `Replicate` feed relatedness and Mendelian checks (plan §9.4). Values refer to `Sample_ID`, `Sample_Name` or `sample_id`, resolved in that order. (src: plan) ### 3.2 IDAT and GTC discovery - **IN-020** `§6.1 · P1` **Recursive IDAT discovery.** - `--idats DIR`, or the RootFolder, is searched recursively for `__{Grn,Red}.idat` and `.idat.gz`. - Matching of the channel token is case-insensitive. - Symlinks are followed, with cycle detection. - Directory listing order MUST NOT influence any result: paths are sorted bytewise. (src: A-G6, E-5) - **IN-021** `§6.1 · P1` **Pairing.** A sample needs exactly one Grn and one Red file. A lone channel → `W_IDAT_UNPAIRED`, and the sample fails with `MISSING_INPUT`. (src: A-G6) - **IN-022** `§6.1 · P1` **Duplicate pairs [DIFF].** - Two or more pairs that resolve to the same `sample_id` are **not allowed**, for example rescans in different subfolders. - Default `--on-duplicate-idat error` → `E_DUP_IDAT` before any processing. It lists every path of every duplicated sample. - `first` picks the bytewise-smallest path, and `newest` the most recent IDAT scan date (field 300), with ties broken by path. Both are deterministic and OS-independent, and log a warning. (src: A-G6, D-28, E-5) - **IN-023** `§6.1 · P1` **IDAT validation.** - Checked: magic, version 3, TOC offsets inside the file, and equal N and address sets in Red and Grn. - Encrypted (version 1) IDATs → `E_IDAT_ENCRYPTED`. - A barcode/position inside the IDAT (402/404) that disagrees with the file name → `W_IDAT_NAME_MISMATCH`, and the file name wins. (src: plan §6.1) - **IN-024** `§6.1, §9 · P1` **IDAT metadata extraction.** For the control table and the report, extract: - imaging date (the scan entry of RunInfo 300), converted to ISO-8601; - scanner ID and scanner software version (RunInfo/scanner fields); - plate (408), well (409), sample ID (406), chip type (403), manifest (401). Missing fields become empty. An IDAT with no scan time → `W_IDAT_NO_SCAN_TIME` (also MET-015). (src: C-G3, B-12) - **IN-025** `§6.1 · P1` **NextSeq 550 IDATs.** - IDATs from NextSeq 550 scans must parse. Different RunInfo layouts and missing fields are tolerated. - Tests use CytoSNP-850K NS550 data when it is available. - QC thresholds carry a caveat for this scanner (MET-018). (src: A-G61, C-G24) - **IN-026** `§14 · v2` **Scan-registration gate [OURS].** - `--scan-metrics FILE` reads the scanner's per-section metrics file (`metrics.txt`). A section with stripe registration < 0.75 is marked `failed`, and its sample is excluded, with reason `scan_registration`. - The status goes into `sample_status.tsv` (CLI-037) and `sample_summary.tsv`. (src: D-26) - **IN-027** `§6, §7.1 · P2` **IDAT ↔ manifest check [DIFF].** Before processing, for each chip type: - compute the fraction of BPM probe addresses present in the IDAT; - coverage < 0.99 → `E_IDAT_MANIFEST_MISMATCH` (exit 2), with the chip type and manifest name; - 0.99 ≤ coverage < 1 → `W_IDAT_LOW_ADDRESS_COVERAGE`; - also compare IDAT field 401/403 with the BPM name, warning only. The reference silently produces bad calls in this case. (src: E-3, D-29) - **IN-030** `§6.5 · P6` **GTC discovery.** - `--gtcs DIR` is searched recursively for `*.gtc`. The sample key is the GTC Sentrix ID (field 1016); for v3/v4 files, the file-name stem. - Duplicate keys → `E_DUP_GTC`. - Sheet matching follows IN-016. (src: A-G6) - **IN-031** `§6.5 · P6` **GTC acceptance.** - `gtc-to-vcf`, `export bedgraph` and QC accept GTC v3–v5 from any producer. A GTC whose `autocall_version` string is not SimurgArray's, or that lacks v5 fields, gets `W_GTC_FOREIGN` once per batch. - `cn call` and `pgx copy-number call|train` need BAF/LRR (v ≥ 4), normalization transforms, and the cluster-file name. Anything else → `E_GTC_UNSUITABLE`. (src: A-G22, B-34) - **IN-032** `§6.5 · P4` **Corrupt GTC [DIFF].** A GTC that fails validation is skipped: bad magic, truncation, out-of-range TOC, or an unreadable string. That sample gets `E_GTC_CORRUPT` at warning level and a `FAILED` summary row; the others continue, and the exit code is 3. The reference aborts the whole run instead. (src: E-2) - **IN-033** `§6.5 · P1` **Tolerant string decoding.** - All .NET 7-bit-length strings are decoded as UTF-8 with replacement characters. - A length prefix that runs past EOF is corruption (IN-032), never an uncaught exception. - Locale-corrupted date strings are preserved as bytes, and decoded for display only. (src: E-1) ### 3.3 Manifest, cluster file and model compatibility - **IN-040** `§6.2 · P1` **BPM.** - Versions 3–5 with locus-entry versions 4/6/7/8 (plan §6.2) are supported. - Manifest names of any length are supported (tested at > 80 chars), as are paths with spaces and non-ASCII characters. - The manifest name and a SHA-256 go into provenance. (src: E-4) - **IN-041** `§6.3 · P1` **CSV manifest.** - `[Heading]`, `[Assay]` and `[Controls]` sections are parsed. - `Assay Format` (e.g. `LCG`, `EX`) is read from `[Heading]`. It supplies the platform default (CLI-032) and the QC profile selection (QC-025). - `SourceSeq` is required for indel REF/ALT resolution. - A missing `[Controls]` or `[Assay]` section, or missing required columns, gives the errors of MET-002 when the CSV is used for QC. (src: A-G23, A-G36, D-24) - **IN-042** `§6.3 · P1` **BPM ↔ CSV consistency [DIFF].** When both are supplied: - locus-name sets must be equal; - counts must be equal; - Chr/MapInfo/SNP/AddressA/AddressB must agree for every locus. Any difference → `E_BPM_CSV_MISMATCH` with counts and the first 10 examples. `--allow-manifest-mismatch` downgrades this to a warning and uses the BPM values. (src: E-3) - **IN-043** `§6.4 · P4` **BPM ↔ EGT consistency [DIFF].** - Every non-control BPM locus name must be present in the EGT. Missing names → `E_EGT_MANIFEST_MISMATCH` (pre-flight, exit 2), with the count and the first 10 names. The reference instead skips each sample with a dictionary-key error. - EGT loci not in the BPM are ignored, with an info count. - A difference between the EGT's `manifest_name` and the BPM name → warning. (src: E-3, B-33) - **IN-044** `§6.4 · P1` **EGT load errors.** Any of the following → `E_EGT_LOAD`, naming the cause: - a corrupt file; - an unsupported `version` or `data_block_version`; - a record-count disagreement. (src: B-33) - **IN-045** `§6.4, §12 · P1` **Cluster-file names.** - Any file name is accepted as input [DIFF]. - ArrayTrain *outputs* whose names contain characters outside `[A-Za-z0-9_-]` get `W_EGT_NAME`, as a portability notice for other tools. (src: A-G17, E-4) - **IN-046** `§6.3, §8.1 · P6` **Manifest mapping conventions** are validated and converted to VCF representation: - positions are 1-based on the plus strand; - PAR loci are reported on X coordinates (chr `XY` → X PAR); - an insertion is anchored at the base immediately 5′ of the insertion; - a deletion is at its most 5′ deleted base; - `Chr=0` or `MapInfo=0` means unmapped. Indels are left-normalised against the FASTA, and the anchor base is added as VCF requires. (src: A-G23, E-19) - **IN-047** `§15 · P1` **Product-file provenance.** Every product file (BPM, CSV, EGT, models, DB, bundle) is recorded in `run_manifest.json` by name, size and SHA-256. Vendor files are never copied into outputs. (src: plan) - **IN-048** `§6 · v2` **Mask file.** See IN-052. **Assay-format platform codes** (`LCG`, `EX`) are stored in model headers (IN-050). (src: A-G36) - **IN-049** `§6, §14 · P4` **Pre-flight check table.** `pipeline plan` and every command run these checks before starting per-sample work. | Check | Severity | Code | Override | |---|---|---|---| | sheet structure, keys, reserved, length, barcode format, names | error | `E_SHEET_*` | – | | duplicate IDAT pairs / GTCs | error | `E_DUP_IDAT`, `E_DUP_GTC` | `--on-duplicate-idat` | | IDAT address coverage vs BPM < 0.99 | error | `E_IDAT_MANIFEST_MISMATCH` | – | | IDAT address coverage < 1.0 | warning | `W_IDAT_LOW_ADDRESS_COVERAGE` | – | | BPM vs CSV mismatch | error | `E_BPM_CSV_MISMATCH` | `--allow-manifest-mismatch` | | BPM loci missing from EGT | error | `E_EGT_MANIFEST_MISMATCH` | – | | EGT manifest name ≠ BPM name | warning | `W_EGT_MANIFEST_NAME` | – | | FASTA/.fai inconsistent | error | `E_FAI_MISMATCH` | – | | FASTA contigs vs manifest chromosomes | error (strict) | `E_CONTIG_MISMATCH` | `--contig-alias auto` | | genome build (manifest vs FASTA lengths) | error | `E_BUILD_MISMATCH` | – | | PGx: alt contig for GSTT1 missing | error | `E_PGX_ALT_CONTIG_MISSING` | – | | CN/cyto model vs manifest probe set | error | `E_MODEL_MANIFEST_MISMATCH` | – | | PGx CN model vs GTC cluster file | error (default) / warning | `E_/W_MODEL_CLUSTER_MISMATCH` | `--cluster-mismatch warn` | | model schema version unsupported | error | `E_MODEL_VERSION` | – | | PGx batch size / passing count | error / warning | `E_PGX_BATCH_TOO_SMALL`, `W_PGX_BATCH_BELOW_RECOMMENDED` | `--min-batch`, `--min-passing` | | star-allele VCFs unsquashed / filtered / unindexed | error | `E_PGX_VCF_UNSUITABLE`, `E_PGX_VCF_UNINDEXED` | `--allow-foreign-vcf` (foreign only) | | cn annotate: sidecar or index missing | error | `E_CN_SIDECAR_MISSING`, `E_CN_VCF_UNINDEXED` | – | (src: A-G63, B-20, E-3) ### 3.4 Models, masks and databases - **IN-050** `§11 · v2` **PGx CN model file** (our own format, `.simurg-pgxcn`). It is a zip that contains: - `header.json`: - `schema_version`, `software_version`; - `manifest_name`, `manifest_revision`, `probe_set_sha256`; - `cluster_file_name`, `cluster_file_sha256`; - `genome_build`, `fasta_contigs_sha256`, `platform` (`LCG|EX`), `mask_sha256`; - `regions_bed_version`, `n_training_samples`, `training_exclusions`, `created`. - `probes.parquet`: per-probe reference statistics. - `regions.bed`. - `mixture.json`: the per-region CN mixture parameters. Compatibility rules: - The manifest probe set must match → otherwise `E_MODEL_MANIFEST_MISMATCH`. - The cluster file must match the one recorded in the input GTCs: - by default the mismatch is an error, `E_MODEL_CLUSTER_MISMATCH`, whose message tells the user to retrain the model (PCN-005); - `--cluster-mismatch warn` downgrades it to `W_MODEL_CLUSTER_MISMATCH`, logging "Cluster file () differs from the CN model's cluster file ()". The reference changed this rule across versions: error, then warning, then error again. Both modes are tested (EDGE-026). (src: A-G37, B-20, D-11, E-39) - **IN-051** `§10 · v2` **Cyto model file** (`.simurg-cyto`). - Contents: `header.json` (schema version, manifest name/revision, probe-set hash, build, FASTA hash, GC window sizes) plus `probes.parquet` (per-probe GC content at 1 kb / 100 kb / 1 Mb, and optional reference LRR statistics). - It is independent of the cluster file, and it is built by us (CYTO-038). - A probe-set mismatch → `E_MODEL_MANIFEST_MISMATCH`. - An older `schema_version` → `E_MODEL_VERSION`, with a rebuild hint. (src: A-G52) - **IN-052** `§11 · v2` **Probe mask (open format).** - A TSV with `probe_namereason`; `#` starts a comment. - It is discovered as `.mask.tsv` next to the BPM, or given with `--mask`. - It is used by PGx CN training and calling, and its SHA-256 is recorded in the model. - A vendor `.msk` file is **not parsed**, because its format is undocumented (OQ-033). (src: A-G35) - **IN-053** `§11 · v2` **PGx DB bundle.** A SQLite file named `pgx_db--.sqlite`, where `sha12` is the first 12 hex digits of the content SHA-256. Its `meta` table holds the source versions (PGX-001), the builder software version and the full SHA-256. See PGX-002. (src: A-G40, D-15) - **IN-054** `§10.4 · v2` **Annotation bundle.** - A versioned zip `simurg-cyto-annot---.zip`, built by `scripts/build_annotation_bundle.sh` from public sources (CYTO-036). - `cn annotate` checks free disk space before extraction, needing ≥ 2 × the bundle's uncompressed size. (src: A-G53) - **IN-055** `§9 · P5` **YAML inputs.** - The extension must be `.yaml` or `.yml` (`E_QC_CONFIG_EXT`). - A file named `*.yaml.txt` or `*.yml.txt` gets a targeted hint about editors appending `.txt`. - Parse errors report the line and column (`E_QC_CONFIG_PARSE`). (src: C-G6) ### 3.5 Reference FASTA and genome cache - **IN-060** `§8 · P6` **FASTA + index.** - The FASTA may be plain or bgzip-compressed; `.fai` (and `.gzi`) are expected next to it. - A missing `.fai` is built into the cache directory (IN-064), never next to a read-only input, with an info message. - A `.fai` whose contig lengths or offsets disagree with the FASTA → `E_FAI_MISMATCH`. The check samples 3 contigs and verifies line lengths. (src: B-33, A-G24) - **IN-061** `§8.2 · P6` **Contig naming.** - Default `--contig-alias strict`: manifest chromosome names (`1`…`22`, `X`, `Y`, `XY`, `MT`) must exist verbatim as FASTA contigs, except that `XY` maps to X. - `auto` applies the alias table `1↔chr1`, `MT↔chrM`, `22_KI270879v1_alt↔chr22_KI270879v1_alt`. - `FILE` loads a two-column alias TSV. - The VCF uses FASTA contig names. (src: A-G24, E-22) - **IN-062** `§11 · v2` **Alt contigs for PGx.** - PGx commands require a GRCh38 FASTA that contains the GSTT1-carrying alt contig (`chr22_KI270879v1_alt` or `22_KI270879v1_alt`). If it is absent → `E_PGX_ALT_CONTIG_MISSING`, with a hint to use an "analysis set with alts". - Genome-wide steps are unaffected by alt contigs: loci on alt contigs are emitted only when the manifest maps probes there. (src: D-10) - **IN-063** `§8 · P6` **Build detection.** - The build is inferred from the chr1 length in the `.fai` (249 250 621 → GRCh37; 248 956 422 → GRCh38) and compared with the manifest `GenomeBuild`. A disagreement → `E_BUILD_MISMATCH`. - Non-human builds are taken from the species profile (IN-065). - Written as `##simurg_genome_build=GRCh38`. (src: B-6) - **IN-064** `§8, §19 · P6` **Genome cache.** - Default: the needed contigs are loaded once into memory (a NumPy uint8 array per contig, about 3.1 GB for human), shared read-only with worker processes through mmap of a cache file at `$SIMURG_CACHE_DIR/genomes//`. - `--no-genome-cache` uses indexed random access through htslib, for low-memory machines or small manifests. - Results MUST be identical in both modes. (src: A-G8) - **IN-065** `§1, §8 · P6` **Species profile.** A YAML file that lists: - build name; - autosome names; - sex-chromosome names; - haploid contigs (e.g. MT); - PAR intervals; - the sex system (XY/ZW/none). `human` (GRCh37, GRCh38) and `bovine_umd3` are built in. It controls ploidy (OUT-018), autosomal call rate, the LRR SD locus set and the sex estimate. (src: C-G22) --- ## 4. Genotyping outputs ### 4.1 GTC - **OUT-001** `§6.5 · P4` **GTC v5 writer.** - All TOC entries of plan §6.5 are written, in ascending ID order. - Strings use .NET 7-bit varint length prefixes and are **ASCII-only** for every field SimurgArray generates. - Date fields (200, 201) are formatted with an invariant culture: `M/d/yyyy h:mm:ss tt`, with ASCII digits and `AM`/`PM`. - The output must be byte-identical under `LC_ALL` = `C`, `tr_TR.UTF-8`, `ar_SA.UTF-8`, `ja_JP.UTF-8` and `de_DE.UTF-8` (EDGE-001). - Non-ASCII sample names from an IDAT are written as UTF-8 with a correct byte-length prefix. (src: E-1) - **OUT-002** `§6.5 · P4` **GTC content rules.** - Field 10 (sample name) is the IDAT name, never the sheet name (IN-011). - Field 100 is the cluster file *name*. Its SHA-256 is recorded in `run_manifest.json` and in the per-sample Parquet, so that CN models can compare it (IN-050). - Field 300 is `SimurgArray `. - Fields 1006–1015 (call rate, gender, LogR dev, p10GC, extended data, BAF, LRR, percentiles) are always filled. (src: A-G16, E-39) - **OUT-003** `§6.5, §16 · P4` **Interoperability.** Written GTCs must parse without warnings in BeadArrayFiles, gtc2vcf and Picard, and must round-trip byte-identically through SimurgArray's own reader and writer. (src: plan) ### 4.2 SNV VCF - **OUT-010** `§8.3, §13 · P6` **File naming.** - The default output is `.snv.vcf.gz` with a `.snv.vcf.gz.tbi` index. The `.snv` infix keeps SNV VCFs apart from `.cyto.cnv.vcf.gz` and `.pgx_cn.vcf.gz` files in the same folder. - With `--no-bgzip`, the output is a plain `.snv.vcf` and no index is written. A warning is logged, because `pgx star-allele call` and `cn annotate` need compressed, indexed input. - `--merge` writes a single `cohort.snv.bcf` with a `.csi` index. (src: A-G25, A-G9) - **OUT-011** `§8.3 · P6` **Exact header** (default options). Lines appear in exactly this order: ``` ##fileformat=VCFv4.3 ##source=SimurgArray ##reference=file:// ##simurg_genome_build= ##simurg_manifest=;sha256= ##simurg_csv_manifest=;sha256= ##simurg_cluster_file= ##simurg_options=unsquash_duplicates=false;exclude_loci=none;include_loci=none;inf1_nocall_informative=false;auxiliary_loci=none;ploidy_policy=sex-aware;contig_alias=strict ##simurg_run= ##contig= … (one line per FASTA contig that carries ≥ 1 manifest locus, in .fai order; identical for all samples of a run) ##FILTER= ##FORMAT= ##FORMAT= ##FORMAT= ##FORMAT= ##SAMPLE=,SentrixID=> #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT ``` - The header contains no `fileDate` and no timestamps, so reruns stay byte-identical (plan §15). - The `##simurg_*` lines record the effective options (CLI-005). - The `##simurg_csv_manifest` line is omitted when no CSV manifest was supplied. - Optional INFO/FORMAT header lines appear only when the matching options are used. (src: B-6, A-G39, plan §8.3) - **OUT-012** `§8.3 · P6` **Record defaults [EMU].** - By default QUAL is `.`, FILTER is `PASS` and INFO is `.`. Call quality is carried only in `FORMAT/GS`. - `--info-fields ASSAY_TYPE,NORM_BIN,GENTRAIN,CLUSTER_SEP,ILMN_STRAND,PROBE_IDS` adds those annotations. This changes plan §8.3, where INFO was always on; it is now opt-in. - `--filter-gs X` adds `##FILTER=` and sets FILTER to `LowGS` when GS < X. (src: B-1, B-2) - **OUT-013** `§8.3 · P6` **Number formatting.** GS uses 4 decimal places (`%.4f`); BAF and LRR use 8 significant digits (`%.8g`). NaN is written as `.`. Formatting is locale-independent and deterministic, using the shortest-round-trip-safe routine from `simurg.core.fmt`. (src: B-7) - **OUT-014** `§8.3 · P6` **Record order.** - Records are sorted by contig in `.fai` order, then POS, then REF, then ALT, then ID. - Output order is identical for every sample of a batch. - The sample column holds the resolved sample name (IN-010). The `##SAMPLE` line maps it to the Sentrix ID. (src: B-6, E-27) - **OUT-015** `§8.1, §8.3 · P6` **REF/ALT assignment.** - REF is always the FASTA base (or sequence) at POS. ALT alleles come **a priori** from the manifest: the probe's alleles are mapped to the plus strand (plan §8.1), with REF removed. They never depend on observed data. - Multiple ALT alleles are listed in order of first appearance, following BPM order of the contributing probes and then allele order within each probe. - When REF is not among a probe's queried alleles, the record is still written and `W_REF_NOT_QUERIED` is raised. ALT then lists all queried alleles, so no sample can be `0/0` for this probe alone. This is typical for MNVs; see OUT-035. (src: B-5, plan §8.3) - **OUT-016** `§8.3 · P6` **BAF/LRR orientation.** - For single-probe biallelic records, BAF is re-oriented so that it refers to ALT. Let A and B be the probe's plus-strand alleles: - if A = REF and B = ALT, write BAF unchanged; - if A = ALT and B = REF, write 1 − BAF. - LRR is never changed. - Test fixtures MUST include a locus where A = ALT (EDGE-017). This intentionally differs from legacy A/B-oriented reports, where BAF may be 1 − BAF. (src: E-21, B §3.3) - **OUT-017** `§8.3 · P6` **Unmerged probes carry full FORMAT.** Single-probe records carry `GT:GS:BAF:LRR` (and extra fields if requested). Merged records carry `GT:GS` only; see OUT-032. (src: B-3) - **OUT-018** `§8.3 · P6` **Ploidy.** With `--ploidy-policy sex-aware` (the default): - Haploid genotypes are written: - on MT for everyone; - on Y and non-PAR X for samples with `sex_est = M`. - Diploid genotypes are written: - on PAR loci (X coordinates) for everyone; - on autosomes. - A heterozygous call in a haploid context is written as `.` (missing) and counted in the audit table as `het_in_haploid`. - Y loci of `F` samples are written as haploid `.` (missing). - `U` samples are treated as diploid on X and haploid on Y/MT, and the header records `unknown_sex_policy=diploid_x`. - `--ploidy-policy diploid` writes diploid genotypes everywhere except MT. (src: E-20, B §3.2) - **OUT-019** `§8.3 · P6` **Skip taxonomy.** Each skipped locus produces one warning (rate-limited, CLI-008) and one audit row. Totals per code appear in the run summary. | Code | Condition | |---|---| | `W_SKIP_UNMAPPED` | `Chr = 0`, or position 0 or missing | | `W_SKIP_INTENSITY_ONLY` | `Intensity_Only = 1` (CN probe; kept in bedGraph LRR) | | `W_SKIP_INDEL_CONTEXT` | indel whose inserted or deleted sequence cannot be determined (no CSV, no SourceSeq, malformed brackets) | | `W_SKIP_INDEL_REF` | indel whose reference allele cannot be determined from the FASTA | | `W_INDEL_PARTIAL_MATCH` | source sequence flanks match the genome only partially (the record is still written if the unambiguous part suffices, otherwise skipped) | | `W_SKIP_EXCLUDED` | probe named in `--exclude-loci` or absent from `--include-loci` | | `W_SKIP_CONTIG` | manifest chromosome absent from the FASTA under the active alias policy | (src: B-5, E-18) - **OUT-020** `§8.1 · P6` **Indels.** - REF and ALT are resolved from the CSV `SourceSeq`/`TopGenomicSeq` against the FASTA, left-normalised and anchored (IN-046). - When the manifest's deletion normalises to an insertion against the reference, or the other way round, the reference-consistent representation is written. The genotype is remapped, and `flipped_indel=true` goes into the audit table. - Duplicate indel probes designed on opposite strands (`[D/I]` and `[I/D]`) whose calls are both heterozygous MUST merge to `0/1`, never `./.`. The reference had this bug until its v1.4; see the regression test in EDGE-012. (src: B-4, E-14, E-19) - **OUT-021** `§8.3 · P6` **SNV and indel at the same position.** - These are written as separate records by default, which is the reference behaviour. - `--merge-snv-indel` merges them into one multi-allelic record using the OUT-031 algorithm. Probes that query different variant types never constrain each other's alleles, except through the allele sets they query. (src: E-15) ### 4.3 Multi-probe merging ("squashing") - **OUT-030** `§8.3 · P6` **Site grouping.** After plus-strand mapping and normalisation, probes whose variant representations share CHROM, POS and REF form one *site*. For indels the normalised span must also be identical. The site's ALT set is the union of all queried non-REF alleles (OUT-015). Duplicate designs of the same variant, and tri-allelic designs, both become one site. (src: B §3.1, E-12) - **OUT-031** `§8.3 · P6` **Genotype combination algorithm** (exact). 1. **Allele classes.** For a probe *p* with queried plus-strand alleles `{a_p, b_p}`, each allele *z* at the site maps to a probe class `c_p(z) ∈ {A, B, ∅}`: - **Infinium II (SNV):** the extended base determines the dye: A/T give red, C/G give green. So `c_p(z) = A` if `z` has the same dye group as `a_p`, and `B` if it has the same dye group as `b_p`. Every base falls into one of the two classes, because Infinium II never queries an A/T or C/G pair. - **Infinium I (SNV), and every indel probe [INF]:** `c_p(z) = A` if `z = a_p`, `B` if `z = b_p`, and `∅` otherwise (no extension, so no signal). 2. **Predicted call.** For a candidate genotype `g`, take the multiset of classes of its alleles (1 allele if haploid, 2 if diploid) and drop the `∅` entries: - `{A}` or `{A,A}` → `AA`; - `{B}` or `{B,B}` → `BB`; - `{A,B}` → `AB`; - empty → `NC` (no signal). 3. **Candidates.** Enumerate every unordered genotype over `{REF} ∪ ALT` with the site's ploidy (OUT-018). 4. **Constrain.** Keep `g` only if, for every probe whose call is not NC, the predicted call equals the observed call. A predicted `NC` never equals an observed call. **An NC probe constrains nothing by default**; it is presumed to be a probe failure. 5. **Optional Infinium-I no-call rule** (`--inf1-nocall-informative`). Let `S` be the survivors of step 4. For each Infinium I probe with an NC call, `S' = {g ∈ S : c_p(z) = ∅ for all z ∈ g}`. The NC is read as "missing signal", so `g` carries none of the probe's queried alleles. - If `S'` is non-empty, `S := S'`. - Otherwise keep `S`, and raise `W_INF1_NC_CONFLICT`, because other assays show the allele. - The exact semantics are [INF]; see OQ-042. 6. **Outcome.** - `|S| = 1` → call it. - `|S| = 0` → `./.` plus `W_MERGE_INCONSISTENT`. - `|S| > 1` → `./.` plus `W_MERGE_AMBIGUOUS`. - All probes NC → `./.` plus `W_MERGE_ALL_NC`. The message text is `Failed to combine genotypes () - (): , …`, with the probes in BPM order. It is also written to the audit table. Required tests (EDGE-011): - Infinium II `[A/G]` = AA with Infinium I `[T/G]` = (T,T) → `./.`, ambiguous between AT and TT. - Infinium I `[T/A]` giving the T-homozygous call, with Infinium II `[A/G]` = NC → `./.`, ambiguous; *not* a T/T call. - A pair that yields 0 survivors → inconsistent. - All probes NC. - The Infinium-I NC option both on and off. (src: B-3, E-12, E-13, A-G20, plan §8.3) - **OUT-032** `§8.3 · P6` **Merged record content.** - FORMAT is `GT:GS` only. BAF and LRR are *omitted* from the FORMAT string, not written as `.`. - GS is the minimum GS over **all** contributing probes, including NC probes [INF, OQ-041]. - QUAL is `.`, FILTER is `PASS` and INFO is `.`, unless `--info-fields` is set. (src: B-3, E-17) - **OUT-033** `§8.3 · P6` **Merged record ID.** The names of all contributing probes, in BPM order, joined by `,` for SNV sites and by `;` for indel sites. (src: B §3.1, E-17) - **OUT-034** `§8.3 · P6` **`--unsquash-duplicates`.** - Each probe gets its own record with full FORMAT, so the site is not merged. - **Exception:** a site whose union allele set is multi-allelic (≥ 2 ALT) is always merged, even with this flag. - The header records `unsquash_duplicates=true`. - For assay validation only; PGx refuses such files (PGX-011). (src: A-G21, E-25) - **OUT-035** `§8.3 · P6` **Auxiliary loci (MNV restoration).** `--auxiliary-loci FILE.vcf` is a VCF whose records have `ID` equal to a manifest probe name, and `CHROM/POS/REF/ALT` equal to the true variant. Validation (each failure → `E_AUX_INVALID` naming the record; exit 2): - (a) exactly one ALT (not multi-allelic); - (b) it is an MNV: `len(REF) = len(ALT) ≥ 2` and REF ≠ ALT; - (c) the probe is the **only** assay at the site (no duplicates); - (d) the bases the probe interrogates match the probe's designed alleles, one each, which fixes the allele mapping: the **first** base of REF and ALT on the plus strand for plus-strand designs (for example, `ATGC[AT/CG]GTAA` designed as `ATGC[A/C]NNNN`: A ↔ AT, C ↔ CG), the **last** base for minus-strand designs, because the probe extends from the other side (as GTCtoVCF); - (e) REF equals the FASTA. The aux record replaces the probe's REF/ALT, and its genotype is mapped through (d). The final report's Top alleles for MNV probes are **never** reverse-complemented by this mapping (EDGE-016). (src: A-G18, E-16) - **OUT-036** `§8.3 · P6` **Locus filter lists.** - `--exclude-loci FILE` (alias `--filter-loci`; the reference's wording "generates" is a documentation error, OQ-002) and `--include-loci FILE` take plain text: one probe name per line, UTF-8, `#` comments, blank lines ignored. - Unknown names produce one warning with a count. - The list's SHA-256 is recorded in the header and in provenance. (src: A-G19) - **OUT-037** `§8.3 · P6` **`--inf1-nocall-informative`** (default off) implements OUT-031 step 5. The user guide MUST state three things: - it raises call rate *and* false positives; - it assumes a cluster file that clusters the "zeroed-out" multi-allelic assays; - it is recorded in the header. (src: A-G20, B-3, E-13) - **OUT-038** `§8.3 · P6` **Index.** - Indexes are written with htslib: tabix `.tbi`, or `.csi` when any contig exceeds 2²⁹ bp. - The index includes the count metadata (the pseudo-bin), so `bcftools index --nrecords` and `--stats` work. - Indexes written by third-party tools are accepted downstream. (src: E-23) - **OUT-039** `§16 · P6` **Validation.** CI runs `bcftools view -H`, `bcftools index --stats` and a VCF-4.3 validator on golden outputs. Any error fails the build. (src: E-24) - **OUT-040** `§8.3 · P6` **Multi-sample BCF** (`--merge`). The same records and FORMAT rules as the per-sample VCFs. Sample order is sheet order, or sorted `sample_id` when there is no sheet. (src: plan) - **OUT-041** `§8.3 · P6` **VCF version.** - SimurgArray writes `VCFv4.3` consistently, with `;`-separated FILTER lists. - It never claims 4.1 while using 4.3 syntax, which is the inconsistency noted in the reference (OQ-044). (src: E-24) - **OUT-042** `§8.2 · P6` **Ambiguous-strand loci.** Plan §8.2 applies unchanged: an ambiguous A/T or C/G locus without RefStrand gets `confidence=low` in the audit. `--drop-ambiguous` removes those loci. (src: plan) - **OUT-043** `§8.4 · P6` **PLINK export.** - `simurg export plink` splits multi-allelic sites into biallelic records by default (`--multiallelic split`, equivalent to `bcftools norm -m-`), or drops them (`--multiallelic drop`). - Merged records with no BAF/LRR are fine for PLINK. - The user guide shows the equivalent manual pipeline: `bcftools merge` → `norm -m-` → `plink2 --vcf`. (src: B-8) ### 4.4 bedGraph - **OUT-050** `§13 · P6` **bedGraph tracks.** Two files per sample: `.baf.bedgraph.gz` and `.lrr.bedgraph.gz` (`--no-gzip` gives plain `.bedgraph`). - **Line 1:** `track type=bedGraph name=" BAF" description="SimurgArray " visibility=full autoScale=off viewLimits=0:1`. For the LRR track, use `viewLimits=-2:2`. - **Rows:** `chromstart0endvalue`, with `start0 = POS − 1` and `end = POS`. Values use `%.6g`. - **Order:** manifest chromosomes in `.fai` order, or manifest order when there is no FASTA; then position. - **Excluded:** - from both tracks: NaN values, unmapped loci and zero-intensity loci; - from BAF only: intensity-only loci (they stay in LRR). - **Probes at the same position:** the mean of their finite values is written, so bedGraph never overlaps [OURS]. - **Orientation:** BAF is written in the manifest A/B orientation. bedGraph has no REF/ALT, and this matches other bedGraph producers. (src: B-32, A-G26) - **OUT-051** `§13 · P6` **bedGraph without re-genotyping.** `export bedgraph` reads GTC + BPM (+ sheet), and does not need an EGT or FASTA. The chromosome naming option `--chrom-style manifest|ucsc` defaults to `manifest`. (src: A-G26) - **OUT-052** `§13 · P6` **No smoothing of raw tracks [DIFF].** - Raw BAF/LRR tracks are never smoothed. - `--smoothed-lrr K` MAY add a *separate* `.lrr_smoothed.bedgraph.gz`: a moving median over 2K+1 probes within each chromosome. - A golden test asserts that the raw track equals the GTC values bit-for-bit after formatting. The reference once corrupted LRR through its smoothing option. (src: E-28, D-3) ### 4.5 Sample summary - **OUT-060** `§13, §9.2 · P4 (QC columns P5)` **`sample_summary.tsv` and `.json`.** One row per sample in the input set: processed, failed and missing samples all get rows. Columns come in this fixed order: | # | Column | Type | Definition | |---|---|---|---| | 1 | `sample_id` | str | `barcode_position` | | 2 | `sample_name` | str | resolved name (IN-010) | | 3 | `sample_folder` | str | IDAT directory relative to the input root | | 4 | `status` | enum | `OK`, `FAILED`, `MISSING_INPUT`, `EXCLUDED_SCAN` | | 5 | `fail_reason` | str | message code + short text; empty if OK | | 6 | `autosomal_call_rate` | float | plan §7.7 | | 7 | `call_rate` | float | plan §7.5 step 10 | | 8 | `logr_dev` | float | plan §7.6 | | 9 | `sex_est` | enum M/F/U | plan §7.8 | | 10 | `sex_reported` | enum M/F/U | IN-014 | | 11 | `sex_check` | enum `match`, `mismatch`, `unknown` | | | 12 | `gc10`, `gc50` | float | plan §7.7 | | 13 | `n_calls`, `n_nocalls`, `n_intensity_only` | int | | | 14 | `p05_x`, `p50_x`, `p95_x`, `p05_y`, `p50_y`, `p95_y` | int | raw intensity percentiles | | 15 | `het_rate` | float | autosomal heterozygosity over calls (P5) | | 16 | `contamination` | float | plan §9.3 (P5; empty if not run) | | 17 | `_norm_r` | float | one column per assay-control probe present (OUT-062) | | 18 | `functional_status`, `control_status`, `control_flag_codes`, `flags` | str | QC-030/031/033 (P5; empty before P5) | | 19… | sheet columns | str | verbatim, sheet order (IN-013) | (src: B-9, A-G14, A-G15, E-11, C-G11, plan §13) - **OUT-061** `§13 · P4` **Failed samples [EMU].** A sample that fails IDAT → GTC still gets a row: - `status=FAILED` and `fail_reason` set; - `autosomal_call_rate = call_rate = 0`; - `logr_dev`, `_norm_r`, `gc10` and `gc50` are NaN (`NaN` in TSV, `null` in JSON); - sex is `U`; - its sheet columns are still passed through. (src: E-2, B-9) - **OUT-062** `§9.1, §13 · P4` **Assay-control locus metric.** - For each probe named in `--assay-control-probes` (default `TGA_Ctrl_5716`, the targeted-amplification control of ePGx arrays) that exists in the manifest, `_norm_r` is the normalized R `xn + yn` of that locus (plan §7.3–§7.4) [INF, OQ-032]. - The column is omitted when no listed probe exists. - Values < 1.0 add the flag `assay_control_fail` (QC-033). (src: A-G32, B-9, C-G10, E-26) - **OUT-063** `§13 · P4` **Precision.** Rates and LRR SD use 8 decimal places (`%.8f`); intensities are integers. NaN is written as `NaN` in TSV. (src: B §2.8) - **OUT-064** `§13 · P4` **JSON form.** `sample_summary.json` is an array of flat objects: same keys and order as the TSV, numbers as JSON numbers, NaN as `null`, sheet values as strings. (src: B-9) - **OUT-065** `§13 · P4` **CSV form.** `--summary-format csv|both` writes `sample_summary.csv` with the same columns, for spreadsheet users. (src: A-G14) - **OUT-066** `§13 · P4` **Compatibility column names.** `--compat-column-names` renames the first columns to reference-style headers: - `Sample ID`, `Sample Name`, `Sample Folder`; - `Autosomal Call Rate`, `Call Rate`, `Log R Ratio Std Dev`; - `Sex Estimate`; - ` Norm R`. It also writes `gt_sample_summary.csv` as an alias file. Our own tools read either form. (src: A-G14, C-G1) ### 4.6 Other genotyping outputs - **OUT-070** `§9.1, §13 · P5` **Control tables.** `simurg genotype` writes `controls_raw.tsv` and `controls_qc.tsv` (QC-009, QC-011) into the batch folder by default. The genotype output folder is therefore a complete QC-report dataset (QC-040). (src: A-G14, C-G1) - **OUT-071** `§6.7 · P6` **Final report export.** `simurg export finalreport` writes a long-format TSV with one row per locus × sample. Columns: - `SNP Name`, `SNP`, `Sample ID`; - `Allele1 - Top`, `Allele2 - Top`, `Allele1 - Forward`, `Allele2 - Forward`, `Allele1 - Plus`, `Allele2 - Plus`; - `GC Score`, `GT Score`, `Log R Ratio`, `B Allele Freq`, `Chr`, `Position`. Allele rules: - Top alleles come from the TOP-strand base calls. - MNV probes are never reverse-complemented (EDGE-016). - For intensity-only loci, BAF is NaN and LRR uses the cluster-mean expected R (plan §7.6). `--matrix` gives a matrix layout. It warns (`W_LARGE_OUTPUT`) when loci × samples > 1e8, with a size estimate. (src: B-10, C-G23, E-16) - **OUT-072** `§7.9 · P6` **Locus summary.** `locus_summary.tsv` has these columns, in order: - `locus_name`, `locus_id`; - `n_nocalls`, `n_calls` (both counted over **samples**), `call_freq`; - `aa_freq`, `ab_freq`, `bb_freq`, `minor_freq`; - `gentrain_score`, `gc50`, `gc10`; - `het_excess`, `chi_test_p100`, `cluster_sep`; - `aa_t_mean`, `aa_t_sd`, `ab_t_mean`, `ab_t_sd`, `bb_t_mean`, `bb_t_sd`; - `aa_r_mean`, `aa_r_sd`, `ab_r_mean`, `ab_r_sd`, `bb_r_mean`, `bb_r_sd`; - `plus_minus_strand` (`+`, `-`, `U`). `het_excess` follows plan §7.9, the k-regularised legacy form. The simpler form `(f_ab − 2pq)/(2pq + ε)` is written as `het_excess_simple`, and OQ-030 records the discrepancy between the two. (src: B-11, C-G23) - **OUT-073** `§15 · P1` **Run manifest.** `run_manifest.json` follows plan §15, plus: - `summary` (CLI-009); - `effective_options` with sources (CLI-005); - `sheet_header` (IN-008); - `skipped_loci_by_code`; - `failed_samples[]`, each with a code; - `schema_versions` of every output. (src: E-47, plan) - **OUT-074** `§8.2, §15 · P6` **Audit tables.** `audit/*.parquet` holds: - one row per harmonisation decision (plan §8.2); - one row per merge decision, with the probes, calls, candidate count and outcome code; - one row per skip (OUT-019); - one row per indel flip (OUT-020). (src: B-3, B-5) - **OUT-075** `§15 · P6` **Schema discipline [DIFF].** - Every JSON and TSV output has a versioned schema (`schema` field in JSON; `#simurg_schema=` comment line at the top of TSVs, which `--no-comment-lines` suppresses). JSON-Schema files are kept under `schemas/`. - CI validates the golden outputs against them. - Numbers are numbers; missing values are `null` in JSON and empty or `NaN` in TSV, as documented per column. - Keys contain no punctuation other than `_`. - A schema change bumps its version and is logged in `CHANGELOG.md` under "Output schema changes". (src: E-48, B-18, B-35) - **OUT-076** `§15 · P6` **Deterministic ordering.** Every output is sorted by documented keys: - samples by input order (sheet order, or sorted `sample_id`); - loci by genome order; - genes by name; - solutions by rank, then lexical tie-breakers. No output depends on hash-map iteration order, directory listing order, the OS or the thread count. Golden tests run on x86-64 and aarch64 (plan U11). (src: E-27, E-5) - **OUT-077** `§15 · P6` **Default-change changelog.** Any change to a default value of any option or threshold is listed in `CHANGELOG.md` under "Default changes", with old → new. The reference changed CNV minimum sizes across minor versions without a machine-readable record. (src: E-47) --- ## 5. QC ### 5.1 Control-probe metrics - **QC-001** `§9.1 · P5` **Control definitions.** - Controls are read from the BPM control config (`addr[:addr…],Category,Color,Name`) or from the CSV `[Controls]` section (`Address,Type,Color_Channel,Name`). - When both manifests are given, the BPM wins, and differences produce one warning. - Each control address gives a raw `green = Grn.Mean[addr]` and `red = Red.Mean[addr]`, using the raw IDAT means with no normalization. (src: A-G27, plan §9.1) - **QC-002** `§9.1 · P5` **Canonical control names.** - Names are normalised before matching: case-fold, collapse whitespace, remove spaces inside parentheses, and treat `Bgnd` and `Bkg` as the same token. - They are then mapped to canonical IDs through `simurg/qc/control_names.yaml`. Examples: `Extension (A)` → `ext_A`; `Biotin (High)` → `biotin_high`; `DNP (Bgnd)` → `dnp_bkg`; `NP (A)` → `np_A`; `String (PM)` → `string_pm`; `NSB (Bgnd) Red` → `nsb_bkg`; `Hyb (High)` → `hyb_high`. - Methylation control names (`BS Conversion I C1`, `GT Mismatch 1 (PM)`, …) are mapped the same way. - When several beads share one canonical ID and a formula uses that probe as a single value, the **mean** is used [OURS]. Formulas that say min or max take the min or max over all matching beads. (src: C-G4) - **QC-003** `§9.1 · P5` **Backgrounds and offset.** - `bkg_grn` = the maximum **green** intensity over `Extension (A)` and `Extension (T)`. - `bkg_red` = the maximum **red** intensity over `Extension (C)` and `Extension (G)`. - Offset `x = 3000`, configurable with `--bg-offset` or YAML `bgOffset`, value ≥ 0. (src: B-13, E-44) - **QC-004** `§9.1 · P5, v2` **Metric catalogue.** There are 24 metrics. Table columns: - **G** = computed for genotyping arrays; **M** = computed for methylation arrays. - **YAML key** is the threshold key used in QC-020. - Every metric uses direction `below`: a value < threshold is flagged. | # | Column (`controls_qc.tsv`) | YAML key | Formula | G | M | |---|---|---|---|---|---| | 1 | `restoration` | `restoration` | Restoration green / (bkg_grn + x) | ✓ | ✓ | | 2 | `staining_green` | `stainingGreen` | Biotin (High) green / Biotin (Bkg) green | ✓ | ✓ | | 3 | `staining_red` | `stainingRed` | DNP (High) red / DNP (Bkg) red | ✓ | ✓ | | 4 | `extension_green` | `extensionGreen` | min(Ext C, Ext G) green / max(Ext A, Ext T) green | ✓ | ✓ | | 5 | `extension_red` | `extensionRed` | min(Ext A, Ext T) red / max(Ext C, Ext G) red | ✓ | ✓ | | 6 | `hyb_high_med` | `hybGreenHighMed` | Hyb (High) green / Hyb (Medium) green | ✓ | ✓ | | 7 | `hyb_med_low` | `hybGreenMediumLow` | Hyb (Medium) green / Hyb (Low) green | ✓ | ✓ | | 8 | `target_removal_1` | `targetRemovalI` | (bkg_grn + x) / Target Removal (1) green | ✓ | ✓ | | 9 | `target_removal_2` | `targetRemovalII` | (bkg_grn + x) / Target Removal 2 green | – | ✓ | | 10 | `stringency` | `stringency` | String (PM) red / String (MM) red (channel configurable, OQ-009) | ✓ | – | | 11 | `nsb_green` | `nonSpecificBindingGreen` | (bkg_grn + x) / max NSB (Bkg) green | ✓ | – | | 12 | `nsb_red` | `nonSpecificBindingRed` | (bkg_red + x) / max NSB (Bkg) red | ✓ | – | | 13 | `np_green` | `npGreen` | min(NP C, NP G) green / max(NP A, NP T) green | ✓ | ✓ | | 14 | `np_red` | `npRed` | min(NP A, NP T) red / max(NP C, NP G) red | ✓ | ✓ | | 15 | `specificity_1_green` | `specificityIGreen` | min(GT Mismatch 1–3 PM) green / max(GT Mismatch 1–3 MM) green | – | ✓ | | 16 | `specificity_1_red` | `specificityIRed` | min(GT Mismatch 4–6 PM) red / max(GT Mismatch 4–6 MM) red | – | ✓ | | 17 | `specificity_2` | `specificityII` | min(Specificity 1–3 red) / max(Specificity 1–3 green) | – | ✓ | | 18 | `specificity_2_bg` | `specificityIIBg` | (bkg_grn + x) / max(Specificity 1–3 green) (OQ-012) | – | ✓ | | 19 | `bs_conv_1_green` | `bisulfiteConversionIGreen` | min(BS Conv I C1, C2) green / max(BS Conv I U1, U2) green | – | ✓ | | 20 | `bs_conv_1_green_bg` | `bisulfiteConversionIGreenBg` | (bkg_grn + x) / max(BS Conv I U1, U2) green | – | ✓ | | 21 | `bs_conv_1_red` | `bisulfiteConversionIRed` | min(BS Conv I C3–C5) red / max(BS Conv I U3–U5) red | – | ✓ | | 22 | `bs_conv_1_red_bg` | `bisulfiteConversionIRedBg` | (bkg_red + x) / max(BS Conv I U3–U5) red (OQ-012) | – | ✓ | | 23 | `bs_conv_2` | `bisulfiteConversionII` | min(BS Conv II 1–4 red) / max(BS Conv II 1–4 green) | – | ✓ | | 24 | `bs_conv_2_bg` | `bisulfiteConversionIIBg` | (bkg_grn + x) / max(BS Conv II 1–4 green) | – | ✓ | Notes: - This corrects plan §9.1, where NSB red must use `bkg_red`. - Restoration and Target Removal II are new relative to the plan. - Genotyping has 13 control metrics plus the assay-control metric (`assayControl`, OUT-062). Methylation has 21 (MET-006). (src: B-13, D-25, C-G5, E-44) - **QC-005** `§9.1 · P5` **Genotyping metric set** = rows 1–8 and 10–14 of QC-004, plus `assay_control_norm_r`, which is copied from the sample summary when present. (src: A-G32, D-25) - **QC-006** `§9.1 · v2` **Methylation metric set** = the 21 rows with M ✓. The same computation code is shared with MET-006. (src: C-G26, D-21) - **QC-007** `§9.1 · P5` **Missing controls.** - Columns that do not apply to the array type are **omitted** from `controls_qc.tsv`. - When a control a metric needs is absent from the manifest, the metric is `NaN` and `W_CTRL_MISSING` is raised once per batch. The warning lists the missing control types or names, for example on custom chips. - These metrics are marked *not evaluable*. They are excluded from pass/fail and listed in `na_metrics` (QC-031). (src: B-31, D-24) - **QC-008** `§9.1 · P5` **Degenerate values.** - A zero denominator raises `W_CTRL_ZERO_DIVISOR`. With a positive numerator the value is `+Inf` (written `Inf`), which passes every `below` check: signal over a perfectly clean background is not a failure (old scanners write background-subtracted intensities of exactly 0; K-021). `0/0` gives `NaN`. - Unlike QC-007, a NaN caused by the *signal* (the probe exists on the chip) counts as out of range, so the sample is flagged with the metric's code. - Bead addresses listed by the manifest but absent from the IDAT are skipped; a control none of whose addresses is on the chip is *missing* (QC-007), not a signal failure. - Negative intensities cannot occur (uint16 input). (src: plan) [OURS] - **QC-009** `§9.1 · P5` **`controls_raw.tsv`** is long format, with one row per control bead per sample: - columns: `sample_id, category, bead_address, control_name, color_label, green_intensity, red_intensity`; - row order: sample order, then manifest control order. `--compat-column-names` writes `controls.raw_metrics.csv` instead, with columns `SampleId, Category, Beadtype, Control, Color, GreenIntensity, RedIntensity`. All controls are written, including NEGATIVE and NORM. The exclusions of QC-010 apply only to the report. (src: B-12, C-G1) - **QC-010** `§9.5 · P5` **Report exclusion list.** Raw-probe views and the QC table's raw-intensity columns leave out: - the categories NEGATIVE and NORM; - the methylation-only `NP (G) 1`–`NP (G) 5`; - `Biotin (5K)` and `DNP (20K)`. Derived metrics are still computed from every probe they need. The list is configurable in `control_names.yaml`. (src: C-G4) - **QC-011** `§9.1 · P5` **`controls_qc.tsv`** has one row per sample. Columns, in order: - `sample_id`, `array_type`; - `imaging_date` (ISO-8601), `sample_plate`, `sample_well`, `scanner_id`, `scanner_version`: taken from the IDAT (IN-024), with plate and well overridden by the sheet; - `bg_offset`; - the applicable metric columns, in QC-004 order. `--compat-column-names` writes `controls.qc_metrics.csv` with `SampleId, ImagingDate, SamplePlate, SampleWell, ScannerId, ScannerVersion, RestorationQC, StainingGreenQC, …`. Its date format is `M/D/YYYY h:mm:ss AM/PM`. (src: B-12, C-G1, C-G3) - **QC-012** `§9, §14 · P5` **Control-only QC path.** `simurg qc controls` produces QC-009 and QC-011 from IDATs plus a manifest. It needs no cluster file and does no genotyping, and it works for both genotyping and methylation arrays. The resulting folder is a valid QC-report dataset with functional QC = `NA`. (src: A-G27, C-G2) - **QC-013** `§9, §13 · P5` **Genotype emits controls.** See OUT-070. (src: A-G14) - **QC-014** `§14 · P5` **Adding controls to an existing genotype folder.** `qc controls --out ` MAY write into a folder that already contains a `sample_summary`, for example from an older SimurgArray version or an external converter, without `--force`, as long as the control tables themselves do not exist yet. The sample IDs must match the summary; unmatched IDs produce a warning. This mirrors the reference's legacy path. (src: C §1.2) ### 5.2 Threshold configuration - **QC-020** `§9 · P5` **Threshold YAML schema.** Two top-level spellings are accepted: - `qc:` → `report:` - `QC:` → `Report:`, for files exported by other tools. ```yaml qc: profile: genotyping # optional; see QC-024 report: callRate: 0.98 # number enables; null (or empty) disables logRDev: 0.20 stainingGreen: 5 stringency: {threshold: 2, direction: below} # mapping form allows direction restoration: null assayControl: 1 # alias: tgaControl bgOffset: 3000 ``` - Keys: `callRate`, `logRDev`, `assayControl`, `bgOffset`, and every YAML key in QC-004. - Aliases: `tgaControl` = `assayControl`; the plan's snake_case `call_rate_min` = `callRate` and `logr_dev_max` = `logRDev`. - `direction` is `below` (flag if value < threshold) or `above` (flag if value > threshold). Defaults are `above` for `logRDev` and `below` for every other key. - A value exactly equal to the threshold passes. (src: A-G30, C-G5, E-44) - **QC-021** `§9 · P5` **Validation.** Each of these errors exits with code 2: - the file must exist (`E_QC_CONFIG_MISSING`: "QC config file was provided but does not exist: "); - the extension must be `.yaml` or `.yml` (`E_QC_CONFIG_EXT`); - it must be valid YAML (`E_QC_CONFIG_PARSE`); - the wrapper `qc`/`QC` → `report`/`Report` must be present (`E_QC_CONFIG_STRUCTURE`); - only known keys are allowed (`E_QC_THRESHOLD_KEY`, which lists the valid keys); - values must be numeric or null (`E_QC_THRESHOLD_VALUE`); - `callRate` must be in [0, 1] and every other threshold ≥ 0 (`E_QC_THRESHOLD_RANGE`). The same checks apply to `--threshold KEY=VALUE` and to YAML uploaded into the HTML report (QC-055). (src: C-G6, C-G21) - **QC-022** `§9 · P5` **Disabled checks.** A `null` or empty value disables a check. Disabled checks are excluded from both status axes, from threshold lines, from heatmap binary colouring and from the table's conditional formatting. (src: A-G30, C 1.8) - **QC-023** `§9 · P5` **Precedence and built-ins.** - Order, highest first: CLI (`--threshold-call-rate`, `--threshold-lrr-sd`, `--threshold KEY=VALUE`), then the `--config` YAML, then `--profile` (or the YAML's `profile:`), then built-in. - Built-in (no profile) [EMU]: `callRate = 0.98` and `logRDev = 0.20` are enabled. **Every other threshold is unset.** - Example: the YAML sets `callRate: 0.90` and the CLI sets 0.95 → 0.95 is used, with source `cli`. - Each effective threshold is recorded with its source (CLI-005, QC-045). (src: A-G30, C-G6) - **QC-024** `§9 · P5, v2` **Shipped profiles** (`simurg/qc/profiles/*.yaml`) are *starting points for review*, not truths. The documentation says they came from the centre and spread of background distributions in a small set of good and failed datasets, and that they should be recalibrated with QC-026. Values where the reference's own documents disagree are listed in OQ-007/OQ-008. | Key | `genotyping` (LCG/HTS) | `genotyping_ex` (EX; provisional, OQ-046) | `genotyping_example` (alternative values) | `methylation_epic` | `methylation_msa` | |---|---|---|---|---|---| | callRate / logRDev | 0.98 / 0.20 | 0.98 / 0.20 | 0.98 / 0.15 | – | – | | stainingGreen / Red | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | | extensionGreen / Red | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | | hybGreenHighMed / MediumLow | 1 / 1 | 1 / 1 | 1.25 / 1.25 | 1 / 1 | 1 / 1 | | targetRemovalI / II | 1 / – | 1 / – | 5 / – | 1 / 1 | 1 / 1 | | restoration | null (1 with `+ffpe`) | null | null | 0 (1 with `+ffpe`) | 0 (1 with `+ffpe`) | | stringency | 2 | 2 | 1.5 | – | – | | nonSpecificBindingGreen / Red | 1 / 1 | 1 / 1 | 5 / 5 | – | – | | npGreen / npRed | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | 2.5 / 3 | | assayControl | 1 | 1 | 1 | – | – | | bisulfiteConversionIGreen / Red | – | – | – | 1 / 1 | 1 / 1 | | bisulfiteConversionIGreenBg / RedBg | – | – | – | 1 / 1 | 0.5 / 0.5 | | bisulfiteConversionII / IIBg | – | – | – | 1 / 1 | 0.5 / 0.5 | | specificityIGreen / Red / II / IIBg | – | – | – | 1 / 1 / 1 / 1 | 1 / 1 / 1 / 1 | | bgOffset | 3000 | 3000 | 3000 | 3000 | 3000 | - The suffix `+ffpe` (e.g. `--profile methylation_epic+ffpe`) sets `restoration: 1`. - Methylation profiles also carry `pvalThreshold: 0.05` (MET-011). (src: C-G7, B-14, D-21, A-G30) - **QC-025** `§9 · P5` **Profile auto-selection.** `--profile auto` picks the profile from the array type (dataset detection, QC-043) and the manifest `Assay Format`: `EX` → `genotyping_ex`, and everything else → `genotyping`. For methylation it goes by manifest name: EPIC → `methylation_epic`, MSA → `methylation_msa`. The report shows which profile was used for each dataset. (src: B-14, C-G7) - **QC-026** `§9 · P5` **Data-driven threshold suggestion.** `simurg qc suggest-thresholds` reads one or more datasets. For every enabled or known metric it computes a robust centre `m` (the median) and a spread `s = 1.4826·MAD` over samples, on the log scale for ratio metrics. - It suggests `exp(m − k·s)` for `below` keys and `exp(m + k·s)` for `above` keys, with k = 4 by default. - It needs ≥ 30 samples per metric; otherwise the metric is left unset, with a note. - Output: a YAML file (QC-020 schema, with comments showing m, s, n and the % flagged at both the current and the suggested value) and a TSV of the same numbers. (src: C-G8) ### 5.3 QC status semantics - **QC-030** `§9.2 · P5` **Functional status.** Values: `PASS`, `FAIL` or `NA`. - `PASS` iff every enabled functional check passes: `autosomal_call_rate ≥ callRate` **and** `logr_dev ≤ logRDev`. - `FAIL` if any enabled check fails. Failed samples (OUT-061) are `FAIL`. - `NA` when no genotyping summary exists: control-only datasets and methylation. - Overall call rate is reported but never drives the status. (src: C-G9, C-G11) - **QC-031** `§9.2 · P5` **Control status.** Values: `CLEAR` or `FLAGGED`. - `FLAGGED` iff any enabled control metric is out of range, including `assayControl`. - `control_flag_codes` lists the failing YAML keys, `;`-joined in QC-004 order (e.g. `npRed;stringency`). - `na_metrics` lists metrics that could not be evaluated (QC-007). - The two status axes are independent: a sample can be `PASS` and `FLAGGED`. The documentation explains how to read the combination: biological or sample failures versus technical assay steps. (src: C-G9, C 1.8) - **QC-032** `§9.2 · P5` **Methylation pass.** `pass_qc = 1` iff every enabled methylation control metric passes; `fail_codes` lists the failures (MET-008). Probe detection rate never affects it unless `--min-prop-passing` is set. (src: D-21, D-22) - **QC-033** `§9.2 · P5` **Sample QC flags** (`flags` column, `;`-joined, informational): - `low_call_rate`, `high_lrr_sd`: mirror the functional checks; - `sex_mismatch`; - `het_outlier` (|z| > 3 within the batch); - `contaminated` (plan §9.3 estimate > 0.02 [OURS]); - `assay_control_fail` (OUT-062); - `duplicate_of:`, `unexpected_relatedness` (plan §9.4). (src: plan §9.2) - **QC-034** `§9.2 · P5` **Assay-control metric in QC.** `assayControl` (default threshold 1 in the genotyping profiles) takes part in control status, in the trend statistics (QC-054) and in the PGx review gate (QC-035). (src: C-G10, A-G32) - **QC-035** `§9.2, §11 · P5` **Recommended downstream gates.** These are documented in the recipes (CLI-040) and applied as `pgx_review` and `cn_training_exclude` flags: - PGx review: `logr_dev > 0.2`, `call_rate < 0.99`, or `assay_control_norm_r < 1.0`. - CN training: the same three conditions (PCN-005). - Cyto: `logr_dev > 0.2` → SampleQuality; `> 0.3` → whole-chromosome-only mode (CYTO-014). (src: A-G65, A-G34) ### 5.4 QC report — inputs, outputs, datasets - **QC-040** `§9.5 · P5` **Input contract.** The report reads **only** dataset folders that contain: - `controls_raw.tsv` and `controls_qc.tsv` (or their compat CSV names): required; - `sample_summary.tsv` (or `.csv`, or the compat `gt_sample_summary.csv`): optional but recommended, because it enables functional QC, call-rate and LRR SD views, heatmaps and metadata. It never touches raw data, so reports can be regenerated or merged later. Joins use `sample_id`. IDs present in one file but not the other → `W_QC_UNMATCHED_SAMPLES`, listing up to 20 of them. Those samples appear with the fields that are available. (src: C-G1, A-G28, C 1.13) - **QC-041** `§9.5 · P5` **`--data` forms.** 1. A folder that contains the required files. 2. A comma list of such folders. 3. A parent folder, searched recursively for every subfolder containing both control files. Discovered folders are sorted by path. Error cases: - A missing path → `E_QC_NO_DATASETS`: "Input data folder does not exist: ". - No valid folder → "No valid dataset folders detected. Each folder must include at least: controls_raw.tsv, controls_qc.tsv". - A listed folder lacking files → `E_QC_DATASET_INCOMPLETE`: "Dataset