diff --git a/CLAUDE.md b/CLAUDE.md index 345f096..30dc343 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -1,6 +1,8 @@ # SimurgArray Personal research toolkit for Illumina Infinium genotyping arrays (IDAT→GTC→VCF, QC, CN, PGx). The full spec and roadmap is SIMURGARRAY_MASTER_PLAN.md — read the relevant section before changing code. +Externally visible behaviour (CLI options, file schemas, QC semantics, messages, edge cases) is defined item-by-item +in SIMURGARRAY_FUNCTIONAL_SPEC.md (IDs like CLI-021, OUT-031); cite item IDs in commits and tests. ## Rules - Work phase by phase (§18). Don't start a phase until the previous gate passes. diff --git a/README.md b/README.md index 420dfdf..b1f9b29 100644 --- a/README.md +++ b/README.md @@ -3,7 +3,8 @@ Personal research toolkit for Illumina Infinium genotyping BeadChip data (IDAT → normalization → genotype calling → BAF/LRR → QC → VCF/PLINK; later CN/LOH, PGx, cluster training). -Start with **SIMURGARRAY_MASTER_PLAN.md** (full specification and roadmap). +Start with **SIMURGARRAY_MASTER_PLAN.md** (architecture, algorithms, roadmap) and +**SIMURGARRAY_FUNCTIONAL_SPEC.md** (416 itemised requirements: CLI, inputs, outputs, QC, CN, PGx, methylation, edge cases). ```bash bash scripts/bootstrap_refs.sh # clone reference open-source repos into refs/ diff --git a/SIMURGARRAY_FUNCTIONAL_SPEC.md b/SIMURGARRAY_FUNCTIONAL_SPEC.md new file mode 100644 index 0000000..561763c --- /dev/null +++ b/SIMURGARRAY_FUNCTIONAL_SPEC.md @@ -0,0 +1,4113 @@ +# SimurgArray — Functional Specification + +> Companion to `SIMURGARRAY_MASTER_PLAN.md` (the "plan"). The plan defines algorithms, architecture, data sources +> and the phased roadmap. **This document defines externally visible behaviour**: commands, options, validation, +> file schemas, QC semantics, messages, edge cases and acceptance tests. Version 1.0 · 2026-09-27. +> +> It consolidates a documentation-level study of a commercial Infinium array-analysis product (the "reference +> product"), whose public documentation was read in full. That study is archived under `_phase0_archive/site_analysis/` +> as files A–E. About 200 suggested additions from their gap analyses were deduplicated and merged here. +> No vendor binary was inspected (plan §2). Documented behaviour is emulated only where it helps users. +> Documentation errors are listed in §12 and are **never** turned into requirements. + +--- + +## 0. Conventions + +### 0.1 Item format +Every requirement has a stable ID, a trace tag and optional source tags: + +`- **CLI-021** \`§14 · P4\` **Title.** Requirement text … (src: A-G14, C-G1)` + +- **ID prefixes:** `SCOPE` scope, `CLI` command line, `IN` inputs, `OUT` genotyping outputs, `QC` control QC and the + QC report, `CYTO` genome-wide CNV/LOH, `PCN` targeted PGx copy number, `PGX` star alleles and phenotypes, `MET` + methylation control QC, `EDGE` edge cases and regression tests, `PERF` performance, `OQ` open questions. +- **Trace tag:** the master-plan section(s), then the roadmap phase (P0–P10, or `v2` for the plan's v2 modules). + The table in §13 is generated from these tags. +- **Source tags** (`src:`) point at gap-analysis items in the archived study. `A-G5` means file A, item G5; + `B-12` means file B, item 12; and so on for C, D and E. `plan` means the item restates or sharpens the plan. + §13.2 checks that every gap item is covered. +- **Keywords:** MUST, SHOULD and MAY as in RFC 2119. +- **Behaviour markers:** + - **[EMU]**: the reference product's documented behaviour, emulated on purpose. + - **[DIFF]**: we deliberately do it differently. The reason is always given. + - **[OURS]**: the reference documents nothing here, so this is SimurgArray's own definition. + - **[INF]** and **[HYP]**: as in plan §0 (our inference; a hypothesis still to be tested). + +### 0.2 Naming conventions (SimurgArray-wide) +- CLI flags: lowercase kebab-case only. No mixed case and no case-sensitive look-alikes. +- File names: lowercase snake_case. Per-sample files are named `..`, where `` is the + *output stem* defined in IN-012. +- TSV/CSV column names and JSON keys: snake_case. Reference-style column names are available only through the + explicit compatibility options (OUT-066, QC-009). +- Numbers are always formatted locale-independently: `.` as the decimal separator and no thousands separators + (EDGE-001). +- Timestamps in our outputs use ISO-8601 UTC (`2026-09-27T10:15:02Z`). Legacy formats are written only inside + GTC fields, for interoperability (OUT-001). +- Genomic coordinates are 1-based and fully closed in every TSV and JSON output. VCF follows the VCF 4.3 + conventions exactly (OUT-041, CYTO-029). + +### 0.3 Relationship to the plan +Where this spec and the plan disagree about **I/O, naming, options or messages**, this spec wins. Where they disagree +about **algorithms or numerics**, the plan wins. Each such conflict found during implementation MUST be logged in +`docs/known_unknowns.md`, and both documents updated. + +--- + +## 1. Scope & compatibility targets + +- **SCOPE-001** `§1 · P0` **Purpose.** SimurgArray reproduces the functional envelope of a modern Infinium + secondary-analysis pipeline, locally and offline: + - IDAT → GTC genotyping with control-probe QC and a sample summary; + - GTC → SNV VCF and bedGraph; + - a multi-dataset interactive QC report; + - genome-wide CNV/LOH/mosaic calling with ISCN annotation (v2); + - targeted PGx copy number and star-allele diplotypes with CPIC/DPWG phenotypes (v2); + - methylation-array control QC (v2). + (src: plan) + +- **SCOPE-002** `§1 · P0` **Emulation targets [EMU].** SimurgArray emulates these reference behaviours, because + users and downstream tools depend on them: + 1. **Workflow shape:** + - genotype → VCF; + - genotype (or control-only QC) → QC report; + - genotype → VCF → targeted CN → star alleles → phenotype; + - genotype → CNV/LOH → annotation. + 2. **Content of the files that move between steps:** + - control raw and QC metric tables; + - the sample summary, with sheet columns passed through; + - the SNV VCF (FORMAT `GT:GS:BAF:LRR`, merged multi-probe records); + - the CNV VCF information content; + - the PGx tables. + 3. **Numeric defaults:** + - GenCall cutoff 0.15; + - QC 0.98 / 0.20; + - per-type CNV size and probe minimums; + - QUAL cap 60 and filters Q10 / Q7; + - PGx batch 24/22 and training 96/90; + - alternative solutions at ≥ 50 % of the top score; + - low-GS review at 0.15. + 4. **Control-probe metric formulas and QC status semantics** (two status axes, `null` disables a check). + 5. **Sample-sheet semantics:** GenomeStudio-compatible sections, case-insensitive columns, name precedence, + RootFolder. + (src: A-G1, C-G1) + +- **SCOPE-003** `§1 · P0` **Deliberate differences [DIFF].** SimurgArray MUST differ from the reference product in + each of the following. Each has its own item; this table is the index. + +| # | Reference behaviour | SimurgArray behaviour | Item | +|---|---|---|---| +| 1 | silently overwrites the output folder | refuses unless `--force`/`--resume` | CLI-010 | +| 2 | duplicate IDAT pairs: one picked, OS-dependent | error `E_DUP_IDAT` (or a deterministic opt-in rule) | IN-022 | +| 3 | CN probe thresholds < 10 silently reset | reset with a warning | CYTO-008 | +| 4 | CNV VCF: `REF=N`, positive SVLEN for DEL, `GT=1/1` placeholder, ID/POS disagreements | VCF-4.3-correct padding-base POS, FASTA REF, signed SVLEN, meaningful GT, consistent IDs; `--compat` modes available | CYTO-029–CYTO-031, PCN-011 | +| 5 | JSON numbers as strings, `"."` sentinels, stray key punctuation | typed JSON, `null`, JSON-Schema validated | OUT-075 | +| 6 | output order and PGx scores differ by OS | byte-identical outputs across OS, CPU and thread count | OUT-076, PGX-029 | +| 7 | licence server; results discarded when over quota | offline; computed results are never discarded | CLI-012, PGX-033 | +| 8 | mixed-case, case-sensitive flags | lowercase kebab-case | CLI-002 | +| 9 | unknown sex → silently male in CN calling | explicit policy + flag | CYTO-024 | +| 10 | no ISCN for LOH/mosaic/≤ 1 kb; first event sometimes unannotated | ISCN for every reported event | CYTO-034 | +| 11 | bedGraph smoothing corrupted LRR | raw tracks are never smoothed | OUT-052 | +| 12 | own tabix writer without count metadata | htslib index with counts | OUT-038 | +| 13 | cluster file names restricted to `[A-Za-z0-9_-]` | any name accepted | IN-045 | +| 14 | BPM/CSV/EGT/IDAT mismatches silent or per-sample | pre-flight validation errors | IN-042–IN-044, IN-027 | +| 15 | semi-custom arrays: PGx must run twice | single run; PGx uses whatever mapped content exists | PGX-030 | +| 16 | per-step inconsistencies (raw vs effective size; GAINLOH counted in some places only) | one consistent rule, documented | CYTO-011, CYTO-033 | + +- **SCOPE-004** `§1.3 · P0` **Out of scope.** The following are not provided and MUST NOT be emulated: + - cloud platform features: workgroups, credits, sharing, notifications, scanner-to-cloud upload, case creation + on interpretation platforms; + - licence servers or quota; + - polyploid genotyping ("diploid only", as in the reference); + - oncology CNV interpretation (`annotation_type` is always `constitutional`); + - novel star-allele discovery. + + The local equivalents of two cloud features *are* in scope as optional v2 conveniences: the scan-registration + gate (IN-026) and the planned-run watcher (CLI-037). (src: A-G13, D-26, D-27) + +- **SCOPE-005** `§1 · P0` **Species and builds.** + - Human GRCh37 and GRCh38 for genotyping, VCF, QC and CNV/LOH. + - PGx is GRCh38-only in v2.0; see PCN-015. + - Non-human diploid organisms are supported for genotyping, GTC, QC and bedGraph, given a user-supplied FASTA and + a *species profile* (IN-065). The bovine UMD3 build (BovineSNP50) is the non-human acceptance case. + - A VCF for a non-human species needs a FASTA whose contig names match the manifest. + (src: C-G22, A-G62) + +- **SCOPE-006** `§16 · P8` **Product test matrix.** Validation (plan §16.5) MUST include at least one dataset per + manifest family: + - genotyping: ASA, CGA, GCRA, GDA, GSA v3, GSA v4 (EX chemistry), JSA, PRS-type (GRCh37 only), BovineSNP50; + - PGx: GDA-ePGx, GSAv4-ePGx, GCRA-ePGx (all GRCh38); + - cyto: GDA-Cyto, GSA-Cyto, CytoSNP-850K (both iScan and NextSeq 550 IDATs); + - methylation: EPIC v1, EPIC v2, MSA. + + Each is run on GRCh37 and GRCh38 where the product supports both. Families with no public data are recorded as + gaps in the validation report. (src: C-G28) + +- **SCOPE-007** `§1 · P10` **Product-file compatibility matrix.** The user guide MUST publish a matrix linking + manifest revision ↔ cluster file ↔ PGx CN model ↔ cyto model ↔ PGx DB ↔ annotation bundle. It lists, for each + combination, which checks are errors and which are warnings (IN-049). `simurg inspect --compat ` prints + the same matrix for the files the user supplies. (src: A-G63) + +- **SCOPE-008** `§3 · P10` **User runtime requirements.** + - **Platforms:** Linux x86-64 and aarch64, macOS arm64. Windows is best-effort (via WSL) and not a release gate. + - **Resources:** + - Genotyping and QC: 4 cores and 8 GB RAM minimum; 8 cores and 32 GB recommended. + - Disk: ≥ 30 GB free for GSA-size batches of 1 000 samples, plus the annotation bundle (CYTO-036). + - PGx CN batches: memory scales with the batch, as documented in PCN-016. + - **Operation:** + - No elevated privileges needed; nothing is written outside the output, cache and tmp directories. + - No network access at run time (offline-first). Downloads happen only in explicit `fetch`/`build` helper scripts. + (src: A-G12, A-G13, A-G43) + +- **SCOPE-009** `§19 · P9` **Batch limits.** There is no hard sample cap. Every command MUST be tested at: + - ≥ 1 152 samples for genotyping, cyto and methylation; + - ≥ 384 samples for PGx; + - 10 000 synthetic samples for genotyping and QC. + + Memory must stay flat in batch size wherever the computation is per-sample. (src: D-31, D-9) + +- **SCOPE-010** `§11 · v2` **Semi-custom and custom arrays.** + - Genotyping, QC, VCF and CNV work on any Infinium genotyping manifest. + - PGx uses every probe that maps to a PGx-DB variant, whichever part of the design it comes from. A second + "commercial-files" run is never required [DIFF]. + - Warnings are issued when a gene's commercial backbone probes are missing (PGX-030). (src: A-G42, D-20) + +--- + +## 2. Command-line interface (`simurg`) + +### 2.1 Structure and global behaviour + +- **CLI-001** `§14 · P1` **Command tree.** Plan §14's flat verbs are canonical. Grouped verbs are subcommands. + ``` + simurg + ├── version # also: simurg --version + ├── inspect ... [--compat] + ├── ingest | normalize # plan §14 (developer/debug steps) + ├── genotype # IDAT → GTC + controls + sample summary (P4) + ├── gtc-to-vcf # GTC → .snv.vcf.gz (P6) + ├── export bedgraph | plink | finalreport | locus-summary (P6) + ├── qc + │ ├── controls # IDAT + manifest → control tables, no cluster file (P5) + │ ├── samples | loci | contamination | relatedness (P5) + │ ├── report # multi-dataset HTML + table (P5) + │ └── suggest-thresholds (P5) + ├── cn # alias: cyto (v2) + │ ├── build-model + │ ├── call + │ └── annotate + ├── pgx (v2) + │ ├── copy-number call | train + │ ├── star-allele call + │ ├── phenotype # alias: annotate + │ ├── coverage + │ └── db build + ├── methyl (v2) + │ ├── qc + │ └── convert-manifest + ├── train # ArrayTrain (plan §12) (v2) + ├── pipeline run | plan | watch (P6; watch v2) + ├── compare (P7) + └── synth (P1+) + ``` + Every group and leaf accepts `--help`. `simurg ` with no leaf prints the group help and exits with 2. + (src: A-G1, A-G44, plan) + +- **CLI-002** `§14 · P1` **Syntax rules.** + - Options are lowercase kebab-case. Boolean options come in `--x/--no-x` pairs where a default-on switch exists. + - List-valued options accept either a comma list with no spaces (`a,b`; surrounding spaces are trimmed with a + warning) or a repeated flag. + - Paths may contain spaces and non-ASCII characters. + - Unknown options are errors (exit 2) and come with a "did you mean" suggestion. + (src: A-G11) + +- **CLI-003** `§14 · P10` **Migration aliases.** A documented table maps reference-style verbs to simurg verbs: + - `genotype call` → `genotype` + - `genotype gtc-to-vcf` → `gtc-to-vcf` + - `genotype gtc-to-bedgraph` → `export bedgraph` + - `qc call` → `qc controls` + - `cyto call|annotate` → `cn call|annotate` + - `pgx copy-number call|train` → identical + - `pgx star-allele call` → identical + - `pgx star-allele annotate` → `pgx phenotype` + + The two-word forms are accepted as hidden aliases, with an informational note. Mixed-case reference flags (for + example the threshold flags) are *not* accepted; the error message names the simurg equivalent. (src: A-G1) + +- **CLI-004** `§14 · P1` **Global options** (accepted by every leaf command). + +| Option | Type | Default | Validation / notes | +|---|---|---|---| +| `--threads N` | int | `0` = auto: physical cores available to the process, respecting CPU affinity and cgroup quota | 0–1024; results MUST NOT depend on N (plan §15) | +| `--log-level` | enum `debug\|info\|warning\|error` | `info` | | +| `--debug` | flag | off | sets `debug` level **and** includes tracebacks in error records | +| `--log-json` (alias `--json-log`) | flag | off | JSON Lines on stderr, schema in CLI-007 | +| `--log-file PATH` | path | `/logs/-.log` | always JSON Lines, regardless of `--log-json` | +| `--quiet` / `--no-progress` | flag | progress bars only when stderr is a TTY | | +| `--seed N` | int | `42` | recorded in the run manifest | +| `--config FILE` | YAML | none | see CLI-019 | +| `--force` | flag | off | allows replacing existing outputs (CLI-010) | +| `--resume` | flag | off | reuses completed per-sample outputs whose input hashes match (plan §14 content-addressed cache) | +| `--tmp-dir DIR` | dir | `/.simurg_tmp` | removed on success | +| `--help`, `--version` | flag | | `--version` prints `SimurgArray +` | + + (src: A-G2, A-G3, A-G7, plan) + +- **CLI-005** `§14, §15 · P1` **Option precedence.** + 1. Explicit CLI option. + 2. The command's section in the `--config` YAML. + 3. The named profile (QC-024), if any. + 4. The built-in default. + + The only environment variables read are `SIMURG_CACHE_DIR` (IN-064) and `NO_COLOR`. The *effective* value of + every option and threshold, with its source (`cli|yaml:|profile:|default`), MUST be recorded: + - in `run_manifest.json`; + - in VCF headers (`##simurg_options=`); + - in JSON outputs (a `parameters` object); + - in the QC table's Thresholds sheet (QC-045). + + (src: A-G30, E-47) + +- **CLI-006** `§14 · P1` **Input-set rule.** Commands that take a sample set accept a folder (`--idats DIR` or + `--gtcs DIR`), a sheet (`--sheet FILE`), or both. + - **Sheet only:** the folder is the sheet's `RootFolder` (IN-008). If it has none → `E_NO_INPUT_FOLDER`. + - **Folder only:** every discovered sample is processed. + - **Both:** the sheet selects and annotates samples; the CLI folder overrides `RootFolder`, with an info message + if they differ. + - **Neither:** `E_NO_INPUT` (exit 2). + (src: A-G5, plan) + +- **CLI-007** `§14 · P1` **Logging.** + - Human-readable logs go to stderr. With `--log-json`, each record is one JSON object per line: + ```json + {"schema":"simurg.log/1","ts":"2026-09-27T10:15:02.123Z","level":"warning", + "code":"W_MERGE_AMBIGUOUS","msg":"Failed to combine genotypes (ambiguous) - rs1 (InfiniumII): AA, rs1_dup (InfiniumI): TT", + "command":"gtc-to-vcf","run_id":"01J9…","step":"merge","sample_id":"204753010023_R01C01", + "locus":"rs1","file":null,"count":1,"elapsed_ms":8123,"exc":null} + ``` + - `exc` holds the formatted traceback only with `--debug`. + - `code` is a stable catalogue ID (CLI-016); `msg` is free text that may change between versions. + - Unknown fields MUST be ignored by consumers. + + (src: A-G2, A-G3) + +- **CLI-008** `§14 · P1` **Warning rate limiting.** + - Each warning code is printed at most 20 times per sample, then once as a summary: `W_… repeated N more times`. + - Every occurrence is still written to the audit tables (plan §8.2), and the run summary counts them per code. + + (src: B-5) + +- **CLI-009** `§14 · P1` **Run summary.** Each command ends with a summary: + - samples succeeded / failed / skipped; + - counts per warning code; + - output paths; + - wall time. + + It goes to stderr and into `run_manifest.json` (`summary` object). (src: E-18, plan) + +- **CLI-010** `§14 · P1` **Overwrite policy [DIFF].** + - If any file the command would write already exists, the command stops before doing any work: `E_OUTPUT_EXISTS` + lists up to 10 paths, exit 2. + - `--force` replaces them. + - `--resume` keeps outputs whose recorded input hashes and parameters match, and recomputes the rest. + - Files not produced by the current command are never deleted. + - Exception: `qc controls` MAY add its control tables to an existing genotype output folder that lacks them + (QC-014). + (src: A-G4, E-6) + +- **CLI-011** `§14 · P1` **Atomic writes.** Every output is written to `.part` in the same directory, then + fsynced and renamed. An interrupted run MUST NOT leave a truncated file under its final name. (src: plan §14) + +- **CLI-012** `§14 · P1` **Never discard results [DIFF].** Per-sample outputs are finalized as soon as each sample + completes. A later failure (another sample, batch step or report) does not remove them. Batch files (summary, QC + tables) are rewritten at the end, and are also written if the run fails part-way. (src: A-G10) + +- **CLI-013** `§14, §15 · P1` **Threading.** + - Parallelism is per sample (a process pool) plus, inside a sample, vectorized/Rust threads (P9). The total never + exceeds `--threads`. + - bgzip compression uses htslib worker threads taken from the same budget. + - Changing `--threads` MUST NOT change any output byte (EDGE-022). + (src: A-G7) + +- **CLI-014** `§14 · P1` **Exit codes.** + +| Code | Meaning | +|---|---| +| 0 | success; warnings are allowed | +| 1 | internal error (a bug). The message asks for a `--debug` rerun | +| 2 | usage or pre-flight validation error; no per-sample work started. Examples: bad option, missing file, schema or mismatch errors, output exists, batch minimum not met | +| 3 | partial failure: ≥ 1 sample failed and ≥ 1 succeeded. Outputs for the successful samples are complete, and the failed samples are listed in the summary | +| 4 | the run completed but no sample succeeded (extension of plan §14) | +| 130 / 143 | interrupted by SIGINT / SIGTERM (CLI-015) | + + (src: A-G10, plan) + +- **CLI-015** `§14 · P1` **Interruption.** On SIGINT/SIGTERM: + 1. Stop scheduling new work. + 2. Let in-flight atomic writes finish or abort them; either way no partial final files. + 3. Write `run_manifest.json` with `status: "interrupted"`. + 4. Exit with 130/143. + + A later `--resume` continues the run. (src: plan) + +- **CLI-016** `§14 · P1` **Message catalogue.** Every error and warning has a stable code. The catalogue + (`docs/messages.md`, generated from `simurg/core/errors.py`) is part of the acceptance tests (EDGE-*). Minimum set: + +| Code | Level | Trigger | +|---|---|---| +| `E_NO_INPUT`, `E_NO_INPUT_FOLDER` | error | CLI-006 | +| `E_OUTPUT_EXISTS` | error | CLI-010 | +| `E_FILE_NOT_FOUND`, `E_FILE_UNREADABLE` | error | any missing or unreadable input, e.g. a manifest path | +| `E_SHEET_*` (`DUP_COLUMN`, `RESERVED`, `TOO_LONG`, `BARCODE_FORMAT`, `NAME_EMPTY`, `NAME_COLLISION`, `DUP_SAMPLE`, `NO_KEY_COLUMNS`, `EMPTY`, `DELIMITER`) | error | IN-001 – IN-016 | +| `W_SHEET_EMPTY_COLUMN`, `W_SHEET_BARCODE_REPAIRED`, `W_SHEET_ROOTFOLDER_OVERRIDDEN` | warning | IN-004, IN-007, IN-008 | +| `E_DUP_IDAT`, `W_IDAT_UNPAIRED`, `E_IDAT_FORMAT`, `E_IDAT_ENCRYPTED`, `W_IDAT_NAME_MISMATCH` | error / warning | IN-020 – IN-023 | +| `E_IDAT_MANIFEST_MISMATCH`, `W_IDAT_LOW_ADDRESS_COVERAGE` | error / warning | IN-027 | +| `E_GTC_CORRUPT` (sample skipped), `W_GTC_FOREIGN`, `E_GTC_UNSUITABLE` | warning / error | IN-031 – IN-033 | +| `E_BPM_CSV_MISMATCH`, `E_EGT_MANIFEST_MISMATCH`, `E_EGT_LOAD`, `W_EGT_NAME` | error / warning | IN-042 – IN-045 | +| `E_FAI_MISMATCH`, `E_CONTIG_MISMATCH`, `E_BUILD_MISMATCH`, `E_PGX_ALT_CONTIG_MISSING` | error | IN-060 – IN-063 | +| `W_REF_NOT_QUERIED`, `W_SKIP_UNMAPPED`, `W_SKIP_INTENSITY_ONLY`, `W_SKIP_INDEL_CONTEXT`, `W_SKIP_INDEL_REF`, `W_INDEL_PARTIAL_MATCH`, `W_SKIP_EXCLUDED` | warning | OUT-019 | +| `W_MERGE_INCONSISTENT`, `W_MERGE_AMBIGUOUS`, `W_MERGE_ALL_NC`, `W_INF1_NC_CONFLICT` | warning | OUT-031 | +| `E_AUX_INVALID` | error | OUT-035 | +| `W_CTRL_MISSING`, `W_CTRL_ZERO_DIVISOR` | warning | QC-007, QC-008 | +| `E_QC_*` (`NO_DATASETS`, `DATASET_INCOMPLETE`, `LABEL_COUNT`, `LABEL_DUPLICATE`, `MIXED_TYPES`, `FORMAT`, `CONFIG_MISSING`, `CONFIG_EXT`, `CONFIG_PARSE`, `CONFIG_STRUCTURE`, `THRESHOLD_KEY`, `THRESHOLD_VALUE`, `THRESHOLD_RANGE`) and `W_QC_LABEL_SANITIZED`, `W_QC_UNMATCHED_SAMPLES` | error / warning | QC-020 – QC-043 | +| `E_MODEL_MANIFEST_MISMATCH`, `E_MODEL_CLUSTER_MISMATCH` / `W_MODEL_CLUSTER_MISMATCH`, `E_MODEL_VERSION` | error / warning | IN-050, IN-051 | +| `E_PGX_BATCH_TOO_SMALL`, `W_PGX_BATCH_BELOW_RECOMMENDED`, `E_PGX_TRAIN_TOO_SMALL` | error / warning | PCN-004, PCN-005 | +| `E_PGX_VCF_UNSUITABLE`, `E_PGX_VCF_UNINDEXED`, `E_PGX_CN_MISSING` | error | PGX-011 | +| `W_CN_PROBE_FLOOR`, `W_CN_THRESHOLD_EXCEEDS_GENOME`, `I_CN_SEX_RESOLVED`, `W_CN_SEX_UNRESOLVED`, `E_CN_SIDECAR_MISSING`, `E_CN_VCF_UNINDEXED` | mixed | CYTO-* | +| `E_METH_*` and `W_METH_*` | mixed | MET-015 | + + (src: B-33, C-G21, A-G56, B-5) + +- **CLI-017** `§14 · P1` **Paths.** + - Relative paths on the CLI resolve against the CWD. Relative paths inside a sheet or config resolve against that + file's directory. + - Symlinks are followed, with loop detection. + - Output paths are printed as absolute paths. + (src: E-4) + +- **CLI-018** `§13 · P1` **Output layout.** + - Default `--layout nested` (plan §13): per-sample files go under `/samples//`, batch files under + `/`. + - `--layout flat` writes everything into `/`, as the reference does. + - Every downstream command discovers inputs recursively, so both layouts work interchangeably. This includes the + star-allele step, which needs SNV and CN VCFs "in one folder" in the reference. (src: A-G44, plan) + +- **CLI-019** `§14 · P1` **Configuration file.** + - `--config` takes YAML with an optional `version: 1`. It has one top-level section per command (`genotype:`, + `gtc_to_vcf:`, `qc:`, `cn:`, `pgx:`, `methyl:`) plus `runtime:`. + - Keys are the snake_case option names. Unknown keys → `E_CONFIG_KEY` (exit 2). + - `pipeline run` uses the same file, extended with `inputs:`, `reference:` and `outputs:` (plan §14 example). In + `outputs:`, `gtc`, `vcf`, `bedgraph`, `plink`, `final_report`, `locus_summary`, `qc_report`: true/false. + - `pipeline plan` warns when `final_report: true` and loci × samples > 1e8, and suggests turning it off (about + 400 GSA-size samples ≈ 50 GB). + (src: D-30, plan) + +### 2.2 Commands + +- **CLI-020** `§14 · P1` **`simurg inspect ...`** + - Prints the header/TOC of IDAT, BPM, CSV manifest, EGT, GTC, sheet, VCF, CN model, PGx DB or annotation bundle. + - `--json` gives machine-readable output. + - `--compat` prints the compatibility matrix for the given files (SCOPE-007). + (src: plan) + +- **CLI-021** `§14 · P4` **`simurg genotype`** (IDAT → GTC + control tables + sample summary). + +| Option | Type | Default | Validation | +|---|---|---|---| +| `--idats DIR` | dir | – | CLI-006; recursive discovery, IN-020 | +| `--sheet FILE` | CSV | – | IN-001 … | +| `--bpm FILE` | BPM | **required** | IN-040 | +| `--egt FILE` | EGT | **required** | IN-043/IN-044 | +| `--csv FILE` | CSV manifest | optional | if given, checked against the BPM (IN-042) and used for control names (QC-001) | +| `--out DIR` | dir | **required** | CLI-010 | +| `--cutoff X` | float | `0.15` | 0 ≤ X ≤ 1; GenCall score < X → no-call | +| `--norm-version` | enum `1.2.0\|1.1.2` | `1.2.0` | plan §7.2 | +| `--exact/--clean` | flag | `--exact` | plan §0 | +| `--gtc/--no-gtc` | flag | on | off keeps calls only in the Parquet output | +| `--controls/--no-controls` | flag | on | writes `controls_raw.tsv`, `controls_qc.tsv` (QC-009, QC-011) | +| `--summary-format` | enum `tsv\|csv\|both` | `tsv` | plus `.json` always | +| `--compat-column-names` | flag | off | OUT-066 | +| `--on-duplicate-idat` | enum `error\|first\|newest` | `error` | IN-022 | +| `--assay-control-probes` | list | `TGA_Ctrl_5716` | OUT-062 | +| `--species-profile` | name/file | `human` | IN-065 | + + Outputs: GTC (OUT-001), `sample_summary.{tsv,json}` (OUT-060), control tables (QC-009/011), `run_manifest.json`. + (src: A-G14, A-G7, plan) + +- **CLI-022** `§14, §8 · P6` **`simurg gtc-to-vcf`.** + +| Option | Type | Default | Validation / notes | +|---|---|---|---| +| `--gtcs DIR` / `--sheet FILE` | dir / CSV | – | CLI-006 | +| `--bpm FILE` | BPM | **required** | | +| `--csv FILE` | CSV manifest | **required** for indels | must contain `SourceSeq`. If absent, indels are skipped with `W_SKIP_INDEL_CONTEXT` and a one-time error-level notice | +| `--fasta FILE` | FASTA | **required** | IN-060 | +| `--out DIR` | dir | **required** | | +| `--no-bgzip` | flag | off | OUT-010 | +| `--genome-cache/--no-genome-cache` | flag | cache on | IN-064 | +| `--auxiliary-loci FILE` | VCF | – | OUT-035 | +| `--exclude-loci FILE` (alias `--filter-loci`) / `--include-loci FILE` | text | – | OUT-036 | +| `--unsquash-duplicates` | flag | off | OUT-034 | +| `--inf1-nocall-informative` | flag | off | OUT-037 | +| `--merge` | flag | off | multi-sample BCF (OUT-040) | +| `--drop-ambiguous` | flag | off | plan §8.2 | +| `--info-fields` | list | none | OUT-012 | +| `--extra-format` | list `X,Y,NORMX,NORMY,THETA,R` | none | plan §8.3 | +| `--filter-gs X` | float | off | adds FILTER `LowGS` when GS < X | +| `--contig-alias` | enum `strict\|auto\|FILE` | `strict` | IN-061 | +| `--ploidy-policy` | enum `sex-aware\|diploid` | `sex-aware` | OUT-018 | +| `--merge-snv-indel` | flag | off | OUT-021 | + + (src: A-G18, A-G19, A-G20, A-G9, A-G8, plan) + +- **CLI-023** `§14 · P6` **`simurg export …`** + - `export bedgraph --gtcs/--sheet --bpm [--no-gzip]`: OUT-050. + - `export plink --vcfs|--gtcs … [--multiallelic split|drop]`: OUT-043. + - `export finalreport --gtcs --bpm --csv [--matrix]`: OUT-071. + - `export locus-summary --gtcs --bpm --egt`: OUT-072. + + None of them re-genotypes: they read GTCs plus the manifest. (src: A-G26) + +- **CLI-024** `§14, §9 · P5` **`simurg qc controls`** (control-only QC, no cluster file). + +| Option | Type | Default | Validation | +|---|---|---|---| +| `--idats DIR` / `--sheet FILE` | | | CLI-006 | +| `--csv FILE` or `--bpm FILE` | manifest | one required | control definitions (QC-001) | +| `--array-type` | enum `genotyping\|methylation` | **required** | selects the metric set (QC-005/QC-006) | +| `--out DIR` | dir | **required** | QC-014 exception to CLI-010 | +| `--bg-offset X` | float | `3000` | ≥ 0 | + + (src: A-G27, C-G2) + +- **CLI-025** `§14, §9 · P5` **`simurg qc samples | loci | contamination | relatedness`.** These follow plan + §9.2–§9.4 and §7.9 and write `sample_qc.tsv`, `locus_summary.tsv`, `contamination.tsv` and `relatedness.tsv`. + They take `--gtcs`/`--sheet`, the manifest and `--out`. `relatedness` also takes `--pedigree-from-sheet` (plan + §9.4). (src: plan) + +- **CLI-026** `§14, §9.5 · P5` **`simurg qc report`.** + +| Option | Type | Default | Validation | +|---|---|---|---| +| `--data` | list of dirs | **required** | one folder, a comma list, or one parent folder (QC-041) | +| `--label` | list | folder names | count must equal the number of datasets (QC-042) | +| `--config FILE` | YAML | none | QC-020/QC-021 | +| `--profile NAME` | enum (QC-024) | none | | +| `--threshold-call-rate X` | float | `0.98` | 0 ≤ X ≤ 1 | +| `--threshold-lrr-sd X` | float | `0.20` | X ≥ 0 | +| `--threshold KEY=VALUE` | repeatable | – | any QC-020 key; `VALUE` may be `null` | +| `--output-format` | enum `xlsx\|csv\|both` | `xlsx` | anything else → `E_QC_FORMAT` | +| `--report-name STEM` | str | `qc_report_YYYY_MM_DD` / `qc_table_YYYY_MM_DD` | QC-044 | +| `--drop-metadata COLS` / `--keep-metadata COLS` / `--no-metadata` | list / flag | keep all | QC-046 | +| `--split-by` | enum `none\|dataset\|plate\|run` | `none` | QC-057 | +| `--out DIR` | dir | CWD | | + + (src: A-G28, A-G29, A-G30, A-G31, C-G6, C-G17, C-G20) + +- **CLI-027** `§9 · P5` **`simurg qc suggest-thresholds --data … [--k 4] [--min-samples 30] --out FILE.yaml`**: + QC-026. (src: C-G8) + +- **CLI-028** `§10 · v2` **`simurg cn call`** (alias `cyto call`). + +| Option | Type | Default | Range / validation | +|---|---|---|---| +| `--gtcs DIR` / `--sheet FILE` | | | CLI-006; GTCs must qualify (IN-031) | +| `--model FILE` | cyto model | **required** | IN-051 | +| `--summary FILE` | sample_summary.tsv | auto-discovered next to the GTCs | source of sex and LRR SD | +| `--fasta FILE` | FASTA | optional; recommended | REF padding base (CYTO-029) | +| `--out DIR` | dir | **required** | | +| `--min-del-probes` / `--min-del-size` | int / kb | 10 / 25 | probes 0–250000 (floored to 10, CYTO-008); size 0–250000 kb | +| `--min-dup-probes` / `--min-dup-size` | int / kb | 10 / 50 | same | +| `--min-loh-probes` / `--min-loh-size` | int / kb | 500 / 3000 | same | +| `--min-gainloh-probes` / `--min-gainloh-size` | int / kb | 300 / 500 | same | +| `--min-mdel-probes` / `--min-mdel-size` | int / kb | 10 / 25 | same | +| `--min-mdup-probes` / `--min-mdup-size` | int / kb | 10 / 50 | same | +| `--min-mloh-probes` / `--min-mloh-size` | int / kb | 500 / 3000 | same | +| `--smoothing K` | int | 5 | 0–1000 probes each side (CYTO-004) | +| `--max-mosaic-fraction F` | float | 1.0 | 0 < F ≤ 1 (CYTO-020) | +| `--min-mosaic-fraction F` | float | 0.10 | 0 ≤ F < 1 (`LowFraction`, CYTO-021) | +| `--filter-qual Q` | int | 10 | 0–60 (FILTER `Q10`) | +| `--cn-cap N` | int | 4 | 3–8 | +| `--gc-correct/--no-gc-correct` | flag | on | | +| `--unknown-sex-policy` | enum `skip-sex-chromosomes\|female\|male\|error` | `skip-sex-chromosomes` | CYTO-024 | +| `--gt-mode` | enum `vcf\|compat` | `vcf` | CYTO-031 | +| `--svlen-mode` | enum `signed\|positive` | `signed` | CYTO-029 | +| `--mask-bed FILE` | BED | built-in centromere/gap mask | CYTO-017 | +| `--no-bgzip` | flag | off | warns that `cn annotate` needs bgzip + index (CLI-039) | + + (src: A-G45, A-G47, A-G48, D-2, D-3, C-G24) + +- **CLI-029** `§10 · v2` **`simurg cn annotate`** (alias `cyto annotate`). + - Required: `--vcfs DIR` (the `cn call` output) and `--annotation-bundle FILE` (CYTO-036). + - Optional: + - the same 14 `--min-*` options, defaulting to the values recorded by `cn call`; they can only tighten, never + loosen; + - `--min-qual Q` (default 20, 0–60); + - `--size-basis effective|raw` (default `effective`, CYTO-011); + - `--iscn-min-size bp` (default 0); + - `--out DIR`. + (src: A-G49, A-G53, A-G54, D-2) + +- **CLI-030** `§10 · v2` **`simurg cn build-model --bpm --fasta [--csv] --out model.simurg-cyto`**: CYTO-038. + (src: A-G52) + +- **CLI-031** `§11 · v2` **`simurg pgx copy-number call`.** + +| Option | Type | Default | Validation | +|---|---|---|---| +| `--gtcs DIR` / `--sheet FILE` | | | GTCs must qualify (IN-031) | +| `--model FILE` | PGx CN model | **required** | IN-050 | +| `--fasta FILE` | GRCh38 FASTA with alt contigs | **required** | IN-062 | +| `--out DIR` | dir | **required** | recommended: the SNV VCF folder (PCN-014) | +| `--min-batch N` | int | 24 | ≥ 1 | +| `--min-passing N` | int | 22 | ≤ `--min-batch` | +| `--recommended-passing N` | int | 96 | warning only | +| `--lrr-sd-max X` | float | 0.20 | passing means LRR SD **<** X (PCN-004) | +| `--cluster-mismatch` | enum `error\|warn` | `error` | IN-050 | +| `--no-bgzip` | flag | off | | + + (src: A-G33, D-11, E-39) + +- **CLI-032** `§11 · v2` **`simurg pgx copy-number train`.** + +| Option | Type | Default | Validation | +|---|---|---|---| +| `--gtcs` / `--sheet` | | | | +| `--bpm FILE` | BPM | **required** | | +| `--csv FILE` | CSV manifest | optional | source of `Assay Format` | +| `--fasta FILE` | FASTA | **required** | | +| `--mask FILE` | TSV | `.mask.tsv` next to the BPM if present | IN-052 | +| `--platform` | enum `auto\|LCG\|EX` | `auto` = from the CSV `Assay Format` heading | error if `auto` and no CSV | +| `--min-samples N` / `--min-passing N` | int | 96 / 90 | | +| `--max-samples N` | int | 3000 | seeded random subset above this, with a note | +| `--train-lrr-sd-max`, `--train-call-rate-min`, `--train-assay-control-min` | float | 0.20 / 0.99 / 1.0 | exclusion rules (PCN-005) | +| `--genome-cache/--no-genome-cache` | flag | on | | +| `--out DIR` | dir | CWD | | + + (src: A-G34, A-G35, A-G36) + +- **CLI-033** `§11 · v2` **`simurg pgx star-allele call`.** + - Required: `--vcfs DIR[,DIR]` (SNV + PGx CN VCFs; recursive; symlinks allowed), `--db FILE` and `--out DIR`. + - Optional: + - `--population NAME` (prior; default `global`); + - `--max-candidates N` (10); + - `--feasible-ratio R` (0.5); + - `--low-gs X` (0.15); + - `--no-cn` (run without CN VCFs; CN-dependent genes are reported as `not_assessed`); + - `--allow-foreign-vcf`. + - Writes `.pgx_calls.parquet` plus preliminary TSVs. + (src: A-G39, A-G41, B-28, E-40) + +- **CLI-034** `§11 · v2` **`simurg pgx phenotype`** (alias `annotate`). + - Required: `--calls DIR`. + - Optional: `--guideline cpic|dpwg` (default `cpic`) and `--out DIR`. + - Writes the TSV set (PGX-026) and `.pgx.json`. It can be re-run with another guideline without + re-calling. (src: A-G41, B-28, D-16) + +- **CLI-035** `§11 · v2` **`simurg pgx db build --sources DIR --manifests BPM[,…] --out FILE`**: PGX-002. + **`simurg pgx coverage --bpm --db`**: PGX-035. (src: A-G40) + +- **CLI-036** `§1.2 · v2` **`simurg methyl qc`.** + - Required: `--idats`/`--sheet`, `--manifest CSV` and `--out`. + - Optional: + - `--profile epic|msa|custom`; + - `--threshold KEY=VALUE`, `--config`; + - `--pval-threshold` (0.05); + - `--bg-offset` (3000); + - `--beta-offset` (0); + - `--min-prop-passing` (off); + - `--pca-probes` (10000); + - `--pca-max-samples-full` (100); + - `--ffpe` (sets restoration to 1). + - **`simurg methyl convert-manifest`**: MET-003. + (src: C-G26, D-21, D-22) + +- **CLI-037** `§14 · P6 (watch: v2)` **`simurg pipeline run|plan|watch config.yaml`.** + - `plan` = dry run: it resolves inputs, validates everything (IN-049) and prints the step DAG, output sizes and + warnings. + - `run` executes, with `--resume` support. + - `watch` (planned-run mode): polls `inputs.idat_dir` until every sheet sample has an IDAT pair; with + `--scan-metrics`, a sample whose scan registration fails counts as failed (IN-026). + - It **aborts** if failed samples > `max_failed_scans` (default 0). + - Otherwise it runs with the passing samples. + - It writes `sample_status.tsv` (`sample_id, status ∈ {ready, failed, pending}, reason`). + (src: D-27, D-30) + +- **CLI-038** `§14 · P7, P1` **`compare`, `synth`, `train`** follow plan §14, §16.4 and §12. (src: plan) + +- **CLI-039** `§14 · P6` **Cross-step guards.** + - Using `--unsquash-duplicates` or an exclude/include list in `gtc-to-vcf` records this in the VCF header; + `pgx star-allele call` refuses such VCFs (PGX-011). + - `pipeline plan` also warns *before* running when the config combines these options with a PGx step. + - `cn call --no-bgzip` warns if the configured pipeline contains `cn annotate`. + (src: A 1.14, A-G39, A-G9) + +- **CLI-040** `§14 · P10` **Workflow recipes** in the user guide, each as copy-pasteable commands plus a + `pipeline` YAML: + 1. genotyping (IDAT → GTC → VCF); + 2. QC report (genotype or control-only, then the report; multi-run trends); + 3. PGx (genotype → VCF → CN → star alleles → phenotype; optional CN training); + 4. cytogenetics (genotype → cn call → cn annotate → bedGraph); + 5. semi-custom arrays (single run, SCOPE-010); + 6. methylation QC. + + Each recipe states the recommended sample gates (QC-035). (src: A-G64, A-G65) + +--- + +## 3. Inputs + +### 3.1 Sample sheet + +- **IN-001** `§6.6 · P1` **Format.** + - The sheet is a CSV file: UTF-8 (a BOM is tolerated), comma-delimited, RFC 4180 quoting, LF or CRLF line endings. + - Optional sections: `[Header]` (key,value rows), `[Manifests]` (**ignored**) and `[Data]` (the table). + - A sheet with no section markers is valid: its first non-empty line is the header row. + - Section names are matched case-insensitively. + - The file is rejected with `E_SHEET_DELIMITER` and a hint in two cases: + - semicolon- or tab-delimited files (e.g. from European Excel); + - JSON sheets. + - An empty `[Data]` section, or a sheet with no rows, → `E_SHEET_EMPTY`. + (src: A-G58, E-8, B-31) + +- **IN-002** `§6.6 · P1` **Key columns.** + - Required: `SentrixBarcode_A` and `SentrixPosition_A`. + - Also accepted, as aliases for the same pair: `beadChipName` and `sampleSectionName`. + - Missing both forms → `E_SHEET_NO_KEY_COLUMNS`. + - Optional recognised columns: + - `Sample_ID`, `Sample_Name`, `Sample_Plate`, `Sample_Well`, `Sample_Group`; + - reported sex: `Gender`, `Sex` or `demographicSex`; + - `Parent1`, `Parent2`, `Replicate`. + (src: A-G59, E-8, plan §6.6) + +- **IN-003** `§6.6 · P1` **Case-insensitive columns.** + - Header names are matched after Unicode case-folding and trimming. + - Two columns that become equal after folding are an error: `E_SHEET_DUP_COLUMN`, with the message "Duplicate + column '' (column names are case-insensitive). Rename or remove one of: , ". + (src: A-G56, E-9) + +- **IN-004** `§6.6 · P1` **Empty header cells.** Columns whose header is empty are ignored, with one + `W_SHEET_EMPTY_COLUMN` warning. Examples: `SentrixBarcode_A,SentrixPosition_A,,` and a single trailing comma. + Such columns MUST NOT count as duplicates of each other, and MUST NOT produce empty-key fields in any output. + (src: E-9) + +- **IN-005** `§6.6 · P1` **Reserved column names.** A sheet column must not collide (case-insensitively) with any + column that SimurgArray itself writes into the sample summary. That covers: + - every OUT-060 column name; + - their reference-style spellings under OUT-066, such as `Sex Estimate`, `Call Rate`, `Log R Ratio Std Dev` and + `Sample Folder`. + + A collision → `E_SHEET_RESERVED`. With `--namespace-sheet-columns`, all pass-through columns are prefixed + `sheet_` instead. (src: A-G56, E-9) + +- **IN-006** `§6.6 · P1` **Length limit.** Column names and cell values are limited to 500 characters (Unicode code + points). A longer one → `E_SHEET_TOO_LONG`, naming the row, the column and its length. (src: A-G56, E-9) + +- **IN-007** `§6.6 · P1` **Barcode and position validation.** + - **Barcode:** after trimming it must match `^[0-9]{10,14}$`. + - Values like `204753010023.0` are repaired to `204753010023`, with `W_SHEET_BARCODE_REPAIRED`. + - Scientific notation (`2.04753E+11`, `2,04753E+11`) cannot be repaired, because precision is lost. It is + rejected with `E_SHEET_BARCODE_FORMAT` and the hint "format the SentrixBarcode_A column as a number with 0 + decimals (or as text) before saving". + - **Position:** must match `^[Rr][0-9]{2}[Cc][0-9]{2}$` and is normalised to upper case. + (src: A-G56, E-9) + +- **IN-008** `§6.6 · P1` **RootFolder.** + - `[Header] RootFolder,` names the IDAT (or GTC) root. A relative path resolves against the sheet's + directory. + - A CLI folder overrides it (`W_SHEET_ROOTFOLDER_OVERRIDDEN`, logged at info level when the two paths are equal). + - Other `[Header]` keys are kept verbatim in `run_manifest.json` under `sheet_header`. + (src: A-G57, E-8) + +- **IN-009** `§6.6 · P1` **Sample identity.** `sample_id` is always `_`. A duplicate + `sample_id` among sheet rows → `E_SHEET_DUP_SAMPLE`. `--on-duplicate-sample drop` MAY instead keep the first row + and warn, as the methylation pipeline of the reference does. (src: B-31, D-24) + +- **IN-010** `§6.6 · P1` **Sample-name precedence [EMU].** + 1. The sheet's `Sample_Name`, if the column exists. + 2. The sample name stored in the IDAT (field 406), or in the GTC (field 10). + 3. `sample_id`. + + Rules for `Sample_Name` when present: + - It must be non-empty in **every** row (`E_SHEET_NAME_EMPTY`). + - It must not equal a *different* sample's `sample_id` (`E_SHEET_NAME_COLLISION`). + - It must be unique across the sheet (`E_SHEET_NAME_COLLISION`) [OURS: required so that output file names are + unique]. + (src: A-G16, E-10) + +- **IN-011** `§6.6 · P1` **Per-command resolution.** + - Names are resolved independently by each command, and the sheet given to that command is the source of truth. + - The name inside a GTC is **never** rewritten: `genotype` stores the IDAT name, or empty. + - A later command given a sheet uses the sheet's names in its outputs: VCF sample column, summaries, JSON. + (src: A-G16, E-10) + +- **IN-012** `§13 · P1` **Output stem.** `` in file names is the resolved sample name after sanitisation: + - characters outside `[A-Za-z0-9._-]` become `_`; + - leading dots are stripped; + - length is capped at 200. + + If two samples sanitise to the same stem → `E_SHEET_NAME_COLLISION`. The unsanitised name is kept inside the files. + (src: plan §13) + +- **IN-013** `§6.6 · P1` **Metadata pass-through.** + - Every sheet column that is not a key column is copied **verbatim** into `sample_summary.{tsv,json}`: the header + as written in the sheet, the values as strings, sheet column order preserved. + - They are placed after the fixed columns (OUT-060). + - The QC report and QC table receive them from the summary (QC-040). + (src: A-G15, E-11, B-9) + +- **IN-014** `§6.6 · P1` **Reported sex.** + - Values from `Gender`, `Sex` or `demographicSex` are normalised: `M`/`Male`/`1` → `M`, `F`/`Female`/`2` → `F`, + everything else (including empty) → `U`. + - If more than one of these columns is present and they disagree → warning, and the first in the order `Sex`, + `Gender`, `demographicSex` wins. + - The result goes into `sex_reported`, and is compared with `sex_est` in `sex_check` (OUT-060). + (src: A-G59) + +- **IN-015** `§6.6 · P1` **Plate, well and group.** + - `Sample_Plate` and `Sample_Well` from the sheet override the IDAT fields 408/409. They drive plate heatmaps + (QC-052) and the plate/well columns of `controls_qc.tsv`. + - `Sample_Group` is the default colour field in plots: control PCA (QC-060) and methylation PCA (MET-013). + (src: C-G3, A-G59, A-G60) + +- **IN-016** `§6.6 · P1` **Sheet vs files.** + - A sheet row with no matching IDAT/GTC gets a summary row with status `MISSING_INPUT` and counts toward exit 3/4. + - Files not listed in the sheet are ignored; an info message gives their count. + (src: plan) + +- **IN-017** `§6.6, §19 · P1` **Scale.** Parsing and matching are O(n) in rows plus files. A 10 000-row sheet + against 10 000 IDAT pairs must parse and match in < 2 s (PERF-007). (src: E-7) + +- **IN-018** `§6.6 · P5` **Parent/replicate columns.** `Parent1`, `Parent2` and `Replicate` feed relatedness and + Mendelian checks (plan §9.4). Values refer to `Sample_ID`, `Sample_Name` or `sample_id`, resolved in that order. + (src: plan) + +### 3.2 IDAT and GTC discovery + +- **IN-020** `§6.1 · P1` **Recursive IDAT discovery.** + - `--idats DIR`, or the RootFolder, is searched recursively for `__{Grn,Red}.idat` and + `.idat.gz`. + - Matching of the channel token is case-insensitive. + - Symlinks are followed, with cycle detection. + - Directory listing order MUST NOT influence any result: paths are sorted bytewise. + (src: A-G6, E-5) + +- **IN-021** `§6.1 · P1` **Pairing.** A sample needs exactly one Grn and one Red file. A lone channel → + `W_IDAT_UNPAIRED`, and the sample fails with `MISSING_INPUT`. (src: A-G6) + +- **IN-022** `§6.1 · P1` **Duplicate pairs [DIFF].** + - Two or more pairs that resolve to the same `sample_id` are **not allowed**, for example rescans in different + subfolders. + - Default `--on-duplicate-idat error` → `E_DUP_IDAT` before any processing. It lists every path of every duplicated + sample. + - `first` picks the bytewise-smallest path, and `newest` the most recent IDAT scan date (field 300), with ties + broken by path. Both are deterministic and OS-independent, and log a warning. + (src: A-G6, D-28, E-5) + +- **IN-023** `§6.1 · P1` **IDAT validation.** + - Checked: magic, version 3, TOC offsets inside the file, and equal N and address sets in Red and Grn. + - Encrypted (version 1) IDATs → `E_IDAT_ENCRYPTED`. + - A barcode/position inside the IDAT (402/404) that disagrees with the file name → `W_IDAT_NAME_MISMATCH`, and the + file name wins. + (src: plan §6.1) + +- **IN-024** `§6.1, §9 · P1` **IDAT metadata extraction.** For the control table and the report, extract: + - imaging date (the scan entry of RunInfo 300), converted to ISO-8601; + - scanner ID and scanner software version (RunInfo/scanner fields); + - plate (408), well (409), sample ID (406), chip type (403), manifest (401). + + Missing fields become empty. An IDAT with no scan time → `W_IDAT_NO_SCAN_TIME` (also MET-015). (src: C-G3, B-12) + +- **IN-025** `§6.1 · P1` **NextSeq 550 IDATs.** + - IDATs from NextSeq 550 scans must parse. Different RunInfo layouts and missing fields are tolerated. + - Tests use CytoSNP-850K NS550 data when it is available. + - QC thresholds carry a caveat for this scanner (MET-018). + (src: A-G61, C-G24) + +- **IN-026** `§14 · v2` **Scan-registration gate [OURS].** + - `--scan-metrics FILE` reads the scanner's per-section metrics file (`metrics.txt`). A section with stripe + registration < 0.75 is marked `failed`, and its sample is excluded, with reason `scan_registration`. + - The status goes into `sample_status.tsv` (CLI-037) and `sample_summary.tsv`. + (src: D-26) + +- **IN-027** `§6, §7.1 · P2` **IDAT ↔ manifest check [DIFF].** Before processing, for each chip type: + - compute the fraction of BPM probe addresses present in the IDAT; + - coverage < 0.99 → `E_IDAT_MANIFEST_MISMATCH` (exit 2), with the chip type and manifest name; + - 0.99 ≤ coverage < 1 → `W_IDAT_LOW_ADDRESS_COVERAGE`; + - also compare IDAT field 401/403 with the BPM name, warning only. + + The reference silently produces bad calls in this case. (src: E-3, D-29) + +- **IN-030** `§6.5 · P6` **GTC discovery.** + - `--gtcs DIR` is searched recursively for `*.gtc`. The sample key is the GTC Sentrix ID (field 1016); for v3/v4 + files, the file-name stem. + - Duplicate keys → `E_DUP_GTC`. + - Sheet matching follows IN-016. + (src: A-G6) + +- **IN-031** `§6.5 · P6` **GTC acceptance.** + - `gtc-to-vcf`, `export bedgraph` and QC accept GTC v3–v5 from any producer. A GTC whose `autocall_version` string + is not SimurgArray's, or that lacks v5 fields, gets `W_GTC_FOREIGN` once per batch. + - `cn call` and `pgx copy-number call|train` need BAF/LRR (v ≥ 4), normalization transforms, and the cluster-file + name. Anything else → `E_GTC_UNSUITABLE`. + (src: A-G22, B-34) + +- **IN-032** `§6.5 · P4` **Corrupt GTC [DIFF].** A GTC that fails validation is skipped: bad magic, truncation, + out-of-range TOC, or an unreadable string. That sample gets `E_GTC_CORRUPT` at warning level and a `FAILED` summary + row; the others continue, and the exit code is 3. The reference aborts the whole run instead. (src: E-2) + +- **IN-033** `§6.5 · P1` **Tolerant string decoding.** + - All .NET 7-bit-length strings are decoded as UTF-8 with replacement characters. + - A length prefix that runs past EOF is corruption (IN-032), never an uncaught exception. + - Locale-corrupted date strings are preserved as bytes, and decoded for display only. + (src: E-1) + +### 3.3 Manifest, cluster file and model compatibility + +- **IN-040** `§6.2 · P1` **BPM.** + - Versions 3–5 with locus-entry versions 4/6/7/8 (plan §6.2) are supported. + - Manifest names of any length are supported (tested at > 80 chars), as are paths with spaces and non-ASCII + characters. + - The manifest name and a SHA-256 go into provenance. + (src: E-4) + +- **IN-041** `§6.3 · P1` **CSV manifest.** + - `[Heading]`, `[Assay]` and `[Controls]` sections are parsed. + - `Assay Format` (e.g. `LCG`, `EX`) is read from `[Heading]`. It supplies the platform default (CLI-032) and the QC + profile selection (QC-025). + - `SourceSeq` is required for indel REF/ALT resolution. + - A missing `[Controls]` or `[Assay]` section, or missing required columns, gives the errors of MET-002 when the + CSV is used for QC. + (src: A-G23, A-G36, D-24) + +- **IN-042** `§6.3 · P1` **BPM ↔ CSV consistency [DIFF].** When both are supplied: + - locus-name sets must be equal; + - counts must be equal; + - Chr/MapInfo/SNP/AddressA/AddressB must agree for every locus. + + Any difference → `E_BPM_CSV_MISMATCH` with counts and the first 10 examples. `--allow-manifest-mismatch` downgrades + this to a warning and uses the BPM values. (src: E-3) + +- **IN-043** `§6.4 · P4` **BPM ↔ EGT consistency [DIFF].** + - Every non-control BPM locus name must be present in the EGT. Missing names → `E_EGT_MANIFEST_MISMATCH` + (pre-flight, exit 2), with the count and the first 10 names. The reference instead skips each sample with a + dictionary-key error. + - EGT loci not in the BPM are ignored, with an info count. + - A difference between the EGT's `manifest_name` and the BPM name → warning. + (src: E-3, B-33) + +- **IN-044** `§6.4 · P1` **EGT load errors.** Any of the following → `E_EGT_LOAD`, naming the cause: + - a corrupt file; + - an unsupported `version` or `data_block_version`; + - a record-count disagreement. + + (src: B-33) + +- **IN-045** `§6.4, §12 · P1` **Cluster-file names.** + - Any file name is accepted as input [DIFF]. + - ArrayTrain *outputs* whose names contain characters outside `[A-Za-z0-9_-]` get `W_EGT_NAME`, as a portability + notice for other tools. + (src: A-G17, E-4) + +- **IN-046** `§6.3, §8.1 · P6` **Manifest mapping conventions** are validated and converted to VCF representation: + - positions are 1-based on the plus strand; + - PAR loci are reported on X coordinates (chr `XY` → X PAR); + - an insertion is anchored at the base immediately 5′ of the insertion; + - a deletion is at its most 5′ deleted base; + - `Chr=0` or `MapInfo=0` means unmapped. + + Indels are left-normalised against the FASTA, and the anchor base is added as VCF requires. (src: A-G23, E-19) + +- **IN-047** `§15 · P1` **Product-file provenance.** Every product file (BPM, CSV, EGT, models, DB, bundle) is + recorded in `run_manifest.json` by name, size and SHA-256. Vendor files are never copied into outputs. (src: plan) + +- **IN-048** `§6 · v2` **Mask file.** See IN-052. **Assay-format platform codes** (`LCG`, `EX`) are stored in model + headers (IN-050). (src: A-G36) + +- **IN-049** `§6, §14 · P4` **Pre-flight check table.** `pipeline plan` and every command run these checks before + starting per-sample work. + +| Check | Severity | Code | Override | +|---|---|---|---| +| sheet structure, keys, reserved, length, barcode format, names | error | `E_SHEET_*` | – | +| duplicate IDAT pairs / GTCs | error | `E_DUP_IDAT`, `E_DUP_GTC` | `--on-duplicate-idat` | +| IDAT address coverage vs BPM < 0.99 | error | `E_IDAT_MANIFEST_MISMATCH` | – | +| IDAT address coverage < 1.0 | warning | `W_IDAT_LOW_ADDRESS_COVERAGE` | – | +| BPM vs CSV mismatch | error | `E_BPM_CSV_MISMATCH` | `--allow-manifest-mismatch` | +| BPM loci missing from EGT | error | `E_EGT_MANIFEST_MISMATCH` | – | +| EGT manifest name ≠ BPM name | warning | `W_EGT_MANIFEST_NAME` | – | +| FASTA/.fai inconsistent | error | `E_FAI_MISMATCH` | – | +| FASTA contigs vs manifest chromosomes | error (strict) | `E_CONTIG_MISMATCH` | `--contig-alias auto` | +| genome build (manifest vs FASTA lengths) | error | `E_BUILD_MISMATCH` | – | +| PGx: alt contig for GSTT1 missing | error | `E_PGX_ALT_CONTIG_MISSING` | – | +| CN/cyto model vs manifest probe set | error | `E_MODEL_MANIFEST_MISMATCH` | – | +| PGx CN model vs GTC cluster file | error (default) / warning | `E_/W_MODEL_CLUSTER_MISMATCH` | `--cluster-mismatch warn` | +| model schema version unsupported | error | `E_MODEL_VERSION` | – | +| PGx batch size / passing count | error / warning | `E_PGX_BATCH_TOO_SMALL`, `W_PGX_BATCH_BELOW_RECOMMENDED` | `--min-batch`, `--min-passing` | +| star-allele VCFs unsquashed / filtered / unindexed | error | `E_PGX_VCF_UNSUITABLE`, `E_PGX_VCF_UNINDEXED` | `--allow-foreign-vcf` (foreign only) | +| cn annotate: sidecar or index missing | error | `E_CN_SIDECAR_MISSING`, `E_CN_VCF_UNINDEXED` | – | + + (src: A-G63, B-20, E-3) + +### 3.4 Models, masks and databases + +- **IN-050** `§11 · v2` **PGx CN model file** (our own format, `.simurg-pgxcn`). It is a zip that contains: + - `header.json`: + - `schema_version`, `software_version`; + - `manifest_name`, `manifest_revision`, `probe_set_sha256`; + - `cluster_file_name`, `cluster_file_sha256`; + - `genome_build`, `fasta_contigs_sha256`, `platform` (`LCG|EX`), `mask_sha256`; + - `regions_bed_version`, `n_training_samples`, `training_exclusions`, `created`. + - `probes.parquet`: per-probe reference statistics. + - `regions.bed`. + - `mixture.json`: the per-region CN mixture parameters. + + Compatibility rules: + - The manifest probe set must match → otherwise `E_MODEL_MANIFEST_MISMATCH`. + - The cluster file must match the one recorded in the input GTCs: + - by default the mismatch is an error, `E_MODEL_CLUSTER_MISMATCH`, whose message tells the user to retrain the + model (PCN-005); + - `--cluster-mismatch warn` downgrades it to `W_MODEL_CLUSTER_MISMATCH`, logging "Cluster file () + differs from the CN model's cluster file ()". + + The reference changed this rule across versions: error, then warning, then error again. Both modes are tested + (EDGE-026). (src: A-G37, B-20, D-11, E-39) + +- **IN-051** `§10 · v2` **Cyto model file** (`.simurg-cyto`). + - Contents: `header.json` (schema version, manifest name/revision, probe-set hash, build, FASTA hash, GC window + sizes) plus `probes.parquet` (per-probe GC content at 1 kb / 100 kb / 1 Mb, and optional reference LRR + statistics). + - It is independent of the cluster file, and it is built by us (CYTO-038). + - A probe-set mismatch → `E_MODEL_MANIFEST_MISMATCH`. + - An older `schema_version` → `E_MODEL_VERSION`, with a rebuild hint. + (src: A-G52) + +- **IN-052** `§11 · v2` **Probe mask (open format).** + - A TSV with `probe_namereason`; `#` starts a comment. + - It is discovered as `.mask.tsv` next to the BPM, or given with `--mask`. + - It is used by PGx CN training and calling, and its SHA-256 is recorded in the model. + - A vendor `.msk` file is **not parsed**, because its format is undocumented (OQ-033). + (src: A-G35) + +- **IN-053** `§11 · v2` **PGx DB bundle.** A SQLite file named `pgx_db--.sqlite`, where `sha12` is + the first 12 hex digits of the content SHA-256. Its `meta` table holds the source versions (PGX-001), the builder + software version and the full SHA-256. See PGX-002. (src: A-G40, D-15) + +- **IN-054** `§10.4 · v2` **Annotation bundle.** + - A versioned zip `simurg-cyto-annot---.zip`, built by `scripts/build_annotation_bundle.sh` + from public sources (CYTO-036). + - `cn annotate` checks free disk space before extraction, needing ≥ 2 × the bundle's uncompressed size. + (src: A-G53) + +- **IN-055** `§9 · P5` **YAML inputs.** + - The extension must be `.yaml` or `.yml` (`E_QC_CONFIG_EXT`). + - A file named `*.yaml.txt` or `*.yml.txt` gets a targeted hint about editors appending `.txt`. + - Parse errors report the line and column (`E_QC_CONFIG_PARSE`). + (src: C-G6) + +### 3.5 Reference FASTA and genome cache + +- **IN-060** `§8 · P6` **FASTA + index.** + - The FASTA may be plain or bgzip-compressed; `.fai` (and `.gzi`) are expected next to it. + - A missing `.fai` is built into the cache directory (IN-064), never next to a read-only input, with an info + message. + - A `.fai` whose contig lengths or offsets disagree with the FASTA → `E_FAI_MISMATCH`. The check samples 3 contigs + and verifies line lengths. + (src: B-33, A-G24) + +- **IN-061** `§8.2 · P6` **Contig naming.** + - Default `--contig-alias strict`: manifest chromosome names (`1`…`22`, `X`, `Y`, `XY`, `MT`) must exist verbatim + as FASTA contigs, except that `XY` maps to X. + - `auto` applies the alias table `1↔chr1`, `MT↔chrM`, `22_KI270879v1_alt↔chr22_KI270879v1_alt`. + - `FILE` loads a two-column alias TSV. + - The VCF uses FASTA contig names. + (src: A-G24, E-22) + +- **IN-062** `§11 · v2` **Alt contigs for PGx.** + - PGx commands require a GRCh38 FASTA that contains the GSTT1-carrying alt contig (`chr22_KI270879v1_alt` or + `22_KI270879v1_alt`). If it is absent → `E_PGX_ALT_CONTIG_MISSING`, with a hint to use an "analysis set with + alts". + - Genome-wide steps are unaffected by alt contigs: loci on alt contigs are emitted only when the manifest maps + probes there. + (src: D-10) + +- **IN-063** `§8 · P6` **Build detection.** + - The build is inferred from the chr1 length in the `.fai` (249 250 621 → GRCh37; 248 956 422 → GRCh38) and + compared with the manifest `GenomeBuild`. A disagreement → `E_BUILD_MISMATCH`. + - Non-human builds are taken from the species profile (IN-065). + - Written as `##simurg_genome_build=GRCh38`. + (src: B-6) + +- **IN-064** `§8, §19 · P6` **Genome cache.** + - Default: the needed contigs are loaded once into memory (a NumPy uint8 array per contig, about 3.1 GB for human), + shared read-only with worker processes through mmap of a cache file at + `$SIMURG_CACHE_DIR/genomes//`. + - `--no-genome-cache` uses indexed random access through htslib, for low-memory machines or small manifests. + - Results MUST be identical in both modes. + (src: A-G8) + +- **IN-065** `§1, §8 · P6` **Species profile.** A YAML file that lists: + - build name; + - autosome names; + - sex-chromosome names; + - haploid contigs (e.g. MT); + - PAR intervals; + - the sex system (XY/ZW/none). + + `human` (GRCh37, GRCh38) and `bovine_umd3` are built in. It controls ploidy (OUT-018), autosomal call rate, the LRR + SD locus set and the sex estimate. (src: C-G22) + +--- + +## 4. Genotyping outputs + +### 4.1 GTC + +- **OUT-001** `§6.5 · P4` **GTC v5 writer.** + - All TOC entries of plan §6.5 are written, in ascending ID order. + - Strings use .NET 7-bit varint length prefixes and are **ASCII-only** for every field SimurgArray generates. + - Date fields (200, 201) are formatted with an invariant culture: `M/d/yyyy h:mm:ss tt`, with ASCII digits and + `AM`/`PM`. + - The output must be byte-identical under `LC_ALL` = `C`, `tr_TR.UTF-8`, `ar_SA.UTF-8`, `ja_JP.UTF-8` and + `de_DE.UTF-8` (EDGE-001). + - Non-ASCII sample names from an IDAT are written as UTF-8 with a correct byte-length prefix. + (src: E-1) + +- **OUT-002** `§6.5 · P4` **GTC content rules.** + - Field 10 (sample name) is the IDAT name, never the sheet name (IN-011). + - Field 100 is the cluster file *name*. Its SHA-256 is recorded in `run_manifest.json` and in the per-sample + Parquet, so that CN models can compare it (IN-050). + - Field 300 is `SimurgArray `. + - Fields 1006–1015 (call rate, gender, LogR dev, p10GC, extended data, BAF, LRR, percentiles) are always filled. + (src: A-G16, E-39) + +- **OUT-003** `§6.5, §16 · P4` **Interoperability.** Written GTCs must parse without warnings in BeadArrayFiles, + gtc2vcf and Picard, and must round-trip byte-identically through SimurgArray's own reader and writer. (src: plan) + +### 4.2 SNV VCF + +- **OUT-010** `§8.3, §13 · P6` **File naming.** + - The default output is `.snv.vcf.gz` with a `.snv.vcf.gz.tbi` index. The `.snv` infix keeps SNV + VCFs apart from `.cyto.cnv.vcf.gz` and `.pgx_cn.vcf.gz` files in the same folder. + - With `--no-bgzip`, the output is a plain `.snv.vcf` and no index is written. A warning is logged, because + `pgx star-allele call` and `cn annotate` need compressed, indexed input. + - `--merge` writes a single `cohort.snv.bcf` with a `.csi` index. + (src: A-G25, A-G9) + +- **OUT-011** `§8.3 · P6` **Exact header** (default options). Lines appear in exactly this order: + ``` + ##fileformat=VCFv4.3 + ##source=SimurgArray + ##reference=file:// + ##simurg_genome_build= + ##simurg_manifest=;sha256= + ##simurg_csv_manifest=;sha256= + ##simurg_cluster_file= + ##simurg_options=unsquash_duplicates=false;exclude_loci=none;include_loci=none;inf1_nocall_informative=false;auxiliary_loci=none;ploidy_policy=sex-aware;contig_alias=strict + ##simurg_run= + ##contig= + … (one line per FASTA contig that carries ≥ 1 manifest locus, in .fai order; identical for all samples of a run) + ##FILTER= + ##FORMAT= + ##FORMAT= + ##FORMAT= + ##FORMAT= + ##SAMPLE=,SentrixID=> + #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT + ``` + - The header contains no `fileDate` and no timestamps, so reruns stay byte-identical (plan §15). + - The `##simurg_*` lines record the effective options (CLI-005). + - The `##simurg_csv_manifest` line is omitted when no CSV manifest was supplied. + - Optional INFO/FORMAT header lines appear only when the matching options are used. + (src: B-6, A-G39, plan §8.3) + +- **OUT-012** `§8.3 · P6` **Record defaults [EMU].** + - By default QUAL is `.`, FILTER is `PASS` and INFO is `.`. Call quality is carried only in `FORMAT/GS`. + - `--info-fields ASSAY_TYPE,NORM_BIN,GENTRAIN,CLUSTER_SEP,ILMN_STRAND,PROBE_IDS` adds those annotations. This + changes plan §8.3, where INFO was always on; it is now opt-in. + - `--filter-gs X` adds `##FILTER=` and sets FILTER to `LowGS` when GS < X. + (src: B-1, B-2) + +- **OUT-013** `§8.3 · P6` **Number formatting.** GS uses 4 decimal places (`%.4f`); BAF and LRR use 8 significant + digits (`%.8g`). NaN is written as `.`. Formatting is locale-independent and deterministic, using the + shortest-round-trip-safe routine from `simurg.core.fmt`. (src: B-7) + +- **OUT-014** `§8.3 · P6` **Record order.** + - Records are sorted by contig in `.fai` order, then POS, then REF, then ALT, then ID. + - Output order is identical for every sample of a batch. + - The sample column holds the resolved sample name (IN-010). The `##SAMPLE` line maps it to the Sentrix ID. + (src: B-6, E-27) + +- **OUT-015** `§8.1, §8.3 · P6` **REF/ALT assignment.** + - REF is always the FASTA base (or sequence) at POS. ALT alleles come **a priori** from the manifest: the probe's + alleles are mapped to the plus strand (plan §8.1), with REF removed. They never depend on observed data. + - Multiple ALT alleles are listed in order of first appearance, following BPM order of the contributing probes and + then allele order within each probe. + - When REF is not among a probe's queried alleles, the record is still written and `W_REF_NOT_QUERIED` is raised. + ALT then lists all queried alleles, so no sample can be `0/0` for this probe alone. This is typical for MNVs; see + OUT-035. + (src: B-5, plan §8.3) + +- **OUT-016** `§8.3 · P6` **BAF/LRR orientation.** + - For single-probe biallelic records, BAF is re-oriented so that it refers to ALT. Let A and B be the probe's + plus-strand alleles: + - if A = REF and B = ALT, write BAF unchanged; + - if A = ALT and B = REF, write 1 − BAF. + - LRR is never changed. + - Test fixtures MUST include a locus where A = ALT (EDGE-017). This intentionally differs from legacy A/B-oriented + reports, where BAF may be 1 − BAF. + (src: E-21, B §3.3) + +- **OUT-017** `§8.3 · P6` **Unmerged probes carry full FORMAT.** Single-probe records carry `GT:GS:BAF:LRR` (and + extra fields if requested). Merged records carry `GT:GS` only; see OUT-032. (src: B-3) + +- **OUT-018** `§8.3 · P6` **Ploidy.** With `--ploidy-policy sex-aware` (the default): + - Haploid genotypes are written: + - on MT for everyone; + - on Y and non-PAR X for samples with `sex_est = M`. + - Diploid genotypes are written: + - on PAR loci (X coordinates) for everyone; + - on autosomes. + - A heterozygous call in a haploid context is written as `.` (missing) and counted in the audit table as + `het_in_haploid`. + - Y loci of `F` samples are written as haploid `.` (missing). + - `U` samples are treated as diploid on X and haploid on Y/MT, and the header records `unknown_sex_policy=diploid_x`. + - `--ploidy-policy diploid` writes diploid genotypes everywhere except MT. + (src: E-20, B §3.2) + +- **OUT-019** `§8.3 · P6` **Skip taxonomy.** Each skipped locus produces one warning (rate-limited, CLI-008) and one + audit row. Totals per code appear in the run summary. + +| Code | Condition | +|---|---| +| `W_SKIP_UNMAPPED` | `Chr = 0`, or position 0 or missing | +| `W_SKIP_INTENSITY_ONLY` | `Intensity_Only = 1` (CN probe; kept in bedGraph LRR) | +| `W_SKIP_INDEL_CONTEXT` | indel whose inserted or deleted sequence cannot be determined (no CSV, no SourceSeq, malformed brackets) | +| `W_SKIP_INDEL_REF` | indel whose reference allele cannot be determined from the FASTA | +| `W_INDEL_PARTIAL_MATCH` | source sequence flanks match the genome only partially (the record is still written if the unambiguous part suffices, otherwise skipped) | +| `W_SKIP_EXCLUDED` | probe named in `--exclude-loci` or absent from `--include-loci` | +| `W_SKIP_CONTIG` | manifest chromosome absent from the FASTA under the active alias policy | + + (src: B-5, E-18) + +- **OUT-020** `§8.1 · P6` **Indels.** + - REF and ALT are resolved from the CSV `SourceSeq`/`TopGenomicSeq` against the FASTA, left-normalised and anchored + (IN-046). + - When the manifest's deletion normalises to an insertion against the reference, or the other way round, the + reference-consistent representation is written. The genotype is remapped, and `flipped_indel=true` goes into the + audit table. + - Duplicate indel probes designed on opposite strands (`[D/I]` and `[I/D]`) whose calls are both heterozygous MUST + merge to `0/1`, never `./.`. The reference had this bug until its v1.4; see the regression test in EDGE-012. + (src: B-4, E-14, E-19) + +- **OUT-021** `§8.3 · P6` **SNV and indel at the same position.** + - These are written as separate records by default, which is the reference behaviour. + - `--merge-snv-indel` merges them into one multi-allelic record using the OUT-031 algorithm. Probes that query + different variant types never constrain each other's alleles, except through the allele sets they query. + (src: E-15) + +### 4.3 Multi-probe merging ("squashing") + +- **OUT-030** `§8.3 · P6` **Site grouping.** After plus-strand mapping and normalisation, probes whose variant + representations share CHROM, POS and REF form one *site*. For indels the normalised span must also be identical. + The site's ALT set is the union of all queried non-REF alleles (OUT-015). Duplicate designs of the same variant, and + tri-allelic designs, both become one site. (src: B §3.1, E-12) + +- **OUT-031** `§8.3 · P6` **Genotype combination algorithm** (exact). + + 1. **Allele classes.** For a probe *p* with queried plus-strand alleles `{a_p, b_p}`, each allele *z* at the site + maps to a probe class `c_p(z) ∈ {A, B, ∅}`: + - **Infinium II (SNV):** the extended base determines the dye: A/T give red, C/G give green. So + `c_p(z) = A` if `z` has the same dye group as `a_p`, and `B` if it has the same dye group as `b_p`. Every base + falls into one of the two classes, because Infinium II never queries an A/T or C/G pair. + - **Infinium I (SNV), and every indel probe [INF]:** `c_p(z) = A` if `z = a_p`, `B` if `z = b_p`, and `∅` + otherwise (no extension, so no signal). + 2. **Predicted call.** For a candidate genotype `g`, take the multiset of classes of its alleles (1 allele if + haploid, 2 if diploid) and drop the `∅` entries: + - `{A}` or `{A,A}` → `AA`; + - `{B}` or `{B,B}` → `BB`; + - `{A,B}` → `AB`; + - empty → `NC` (no signal). + 3. **Candidates.** Enumerate every unordered genotype over `{REF} ∪ ALT` with the site's ploidy (OUT-018). + 4. **Constrain.** Keep `g` only if, for every probe whose call is not NC, the predicted call equals the observed + call. A predicted `NC` never equals an observed call. **An NC probe constrains nothing by default**; it is + presumed to be a probe failure. + 5. **Optional Infinium-I no-call rule** (`--inf1-nocall-informative`). Let `S` be the survivors of step 4. For each + Infinium I probe with an NC call, `S' = {g ∈ S : c_p(z) = ∅ for all z ∈ g}`. The NC is read as "missing signal", + so `g` carries none of the probe's queried alleles. + - If `S'` is non-empty, `S := S'`. + - Otherwise keep `S`, and raise `W_INF1_NC_CONFLICT`, because other assays show the allele. + - The exact semantics are [INF]; see OQ-042. + 6. **Outcome.** + - `|S| = 1` → call it. + - `|S| = 0` → `./.` plus `W_MERGE_INCONSISTENT`. + - `|S| > 1` → `./.` plus `W_MERGE_AMBIGUOUS`. + - All probes NC → `./.` plus `W_MERGE_ALL_NC`. + + The message text is `Failed to combine genotypes () - (): , …`, + with the probes in BPM order. It is also written to the audit table. + + Required tests (EDGE-011): + - Infinium II `[A/G]` = AA with Infinium I `[T/G]` = (T,T) → `./.`, ambiguous between AT and TT. + - Infinium I `[T/A]` giving the T-homozygous call, with Infinium II `[A/G]` = NC → `./.`, ambiguous; *not* a T/T + call. + - A pair that yields 0 survivors → inconsistent. + - All probes NC. + - The Infinium-I NC option both on and off. + + (src: B-3, E-12, E-13, A-G20, plan §8.3) + +- **OUT-032** `§8.3 · P6` **Merged record content.** + - FORMAT is `GT:GS` only. BAF and LRR are *omitted* from the FORMAT string, not written as `.`. + - GS is the minimum GS over **all** contributing probes, including NC probes [INF, OQ-041]. + - QUAL is `.`, FILTER is `PASS` and INFO is `.`, unless `--info-fields` is set. + (src: B-3, E-17) + +- **OUT-033** `§8.3 · P6` **Merged record ID.** The names of all contributing probes, in BPM order, joined by `,` for + SNV sites and by `;` for indel sites. (src: B §3.1, E-17) + +- **OUT-034** `§8.3 · P6` **`--unsquash-duplicates`.** + - Each probe gets its own record with full FORMAT, so the site is not merged. + - **Exception:** a site whose union allele set is multi-allelic (≥ 2 ALT) is always merged, even with this flag. + - The header records `unsquash_duplicates=true`. + - For assay validation only; PGx refuses such files (PGX-011). + (src: A-G21, E-25) + +- **OUT-035** `§8.3 · P6` **Auxiliary loci (MNV restoration).** `--auxiliary-loci FILE.vcf` is a VCF whose records + have `ID` equal to a manifest probe name, and `CHROM/POS/REF/ALT` equal to the true variant. + + Validation (each failure → `E_AUX_INVALID` naming the record; exit 2): + - (a) exactly one ALT (not multi-allelic); + - (b) it is an MNV: `len(REF) = len(ALT) ≥ 2` and REF ≠ ALT; + - (c) the probe is the **only** assay at the site (no duplicates); + - (d) the first bases, on the plus strand, of REF and ALT match the probe's designed alleles, one each (for + example, `ATGC[AT/CG]GTAA` designed as `ATGC[A/C]NNNN`: A ↔ AT, C ↔ CG), which fixes the allele mapping; + - (e) REF equals the FASTA. + + The aux record replaces the probe's REF/ALT, and its genotype is mapped through (d). The final report's Top alleles + for MNV probes are **never** reverse-complemented by this mapping (EDGE-016). (src: A-G18, E-16) + +- **OUT-036** `§8.3 · P6` **Locus filter lists.** + - `--exclude-loci FILE` (alias `--filter-loci`; the reference's wording "generates" is a documentation error, + OQ-002) and `--include-loci FILE` take plain text: one probe name per line, UTF-8, `#` comments, blank lines + ignored. + - Unknown names produce one warning with a count. + - The list's SHA-256 is recorded in the header and in provenance. + (src: A-G19) + +- **OUT-037** `§8.3 · P6` **`--inf1-nocall-informative`** (default off) implements OUT-031 step 5. The user guide + MUST state three things: + - it raises call rate *and* false positives; + - it assumes a cluster file that clusters the "zeroed-out" multi-allelic assays; + - it is recorded in the header. + + (src: A-G20, B-3, E-13) + +- **OUT-038** `§8.3 · P6` **Index.** + - Indexes are written with htslib: tabix `.tbi`, or `.csi` when any contig exceeds 2²⁹ bp. + - The index includes the count metadata (the pseudo-bin), so `bcftools index --nrecords` and `--stats` work. + - Indexes written by third-party tools are accepted downstream. + (src: E-23) + +- **OUT-039** `§16 · P6` **Validation.** CI runs `bcftools view -H`, `bcftools index --stats` and a VCF-4.3 + validator on golden outputs. Any error fails the build. (src: E-24) + +- **OUT-040** `§8.3 · P6` **Multi-sample BCF** (`--merge`). The same records and FORMAT rules as the per-sample + VCFs. Sample order is sheet order, or sorted `sample_id` when there is no sheet. (src: plan) + +- **OUT-041** `§8.3 · P6` **VCF version.** + - SimurgArray writes `VCFv4.3` consistently, with `;`-separated FILTER lists. + - It never claims 4.1 while using 4.3 syntax, which is the inconsistency noted in the reference (OQ-044). + (src: E-24) + +- **OUT-042** `§8.2 · P6` **Ambiguous-strand loci.** Plan §8.2 applies unchanged: an ambiguous A/T or C/G locus + without RefStrand gets `confidence=low` in the audit. `--drop-ambiguous` removes those loci. (src: plan) + +- **OUT-043** `§8.4 · P6` **PLINK export.** + - `simurg export plink` splits multi-allelic sites into biallelic records by default (`--multiallelic split`, + equivalent to `bcftools norm -m-`), or drops them (`--multiallelic drop`). + - Merged records with no BAF/LRR are fine for PLINK. + - The user guide shows the equivalent manual pipeline: `bcftools merge` → `norm -m-` → `plink2 --vcf`. + (src: B-8) + +### 4.4 bedGraph + +- **OUT-050** `§13 · P6` **bedGraph tracks.** Two files per sample: `.baf.bedgraph.gz` and + `.lrr.bedgraph.gz` (`--no-gzip` gives plain `.bedgraph`). + - **Line 1:** `track type=bedGraph name=" BAF" description="SimurgArray " visibility=full autoScale=off viewLimits=0:1`. + For the LRR track, use `viewLimits=-2:2`. + - **Rows:** `chromstart0endvalue`, with `start0 = POS − 1` and `end = POS`. Values use `%.6g`. + - **Order:** manifest chromosomes in `.fai` order, or manifest order when there is no FASTA; then position. + - **Excluded:** + - from both tracks: NaN values, unmapped loci and zero-intensity loci; + - from BAF only: intensity-only loci (they stay in LRR). + - **Probes at the same position:** the mean of their finite values is written, so bedGraph never overlaps [OURS]. + - **Orientation:** BAF is written in the manifest A/B orientation. bedGraph has no REF/ALT, and this matches other + bedGraph producers. + + (src: B-32, A-G26) + +- **OUT-051** `§13 · P6` **bedGraph without re-genotyping.** `export bedgraph` reads GTC + BPM (+ sheet), and does not + need an EGT or FASTA. The chromosome naming option `--chrom-style manifest|ucsc` defaults to `manifest`. + (src: A-G26) + +- **OUT-052** `§13 · P6` **No smoothing of raw tracks [DIFF].** + - Raw BAF/LRR tracks are never smoothed. + - `--smoothed-lrr K` MAY add a *separate* `.lrr_smoothed.bedgraph.gz`: a moving median over 2K+1 probes + within each chromosome. + - A golden test asserts that the raw track equals the GTC values bit-for-bit after formatting. The reference once + corrupted LRR through its smoothing option. + (src: E-28, D-3) + +### 4.5 Sample summary + +- **OUT-060** `§13, §9.2 · P4 (QC columns P5)` **`sample_summary.tsv` and `.json`.** One row per sample in the input + set: processed, failed and missing samples all get rows. Columns come in this fixed order: + +| # | Column | Type | Definition | +|---|---|---|---| +| 1 | `sample_id` | str | `barcode_position` | +| 2 | `sample_name` | str | resolved name (IN-010) | +| 3 | `sample_folder` | str | IDAT directory relative to the input root | +| 4 | `status` | enum | `OK`, `FAILED`, `MISSING_INPUT`, `EXCLUDED_SCAN` | +| 5 | `fail_reason` | str | message code + short text; empty if OK | +| 6 | `autosomal_call_rate` | float | plan §7.7 | +| 7 | `call_rate` | float | plan §7.5 step 10 | +| 8 | `logr_dev` | float | plan §7.6 | +| 9 | `sex_est` | enum M/F/U | plan §7.8 | +| 10 | `sex_reported` | enum M/F/U | IN-014 | +| 11 | `sex_check` | enum `match`, `mismatch`, `unknown` | | +| 12 | `gc10`, `gc50` | float | plan §7.7 | +| 13 | `n_calls`, `n_nocalls`, `n_intensity_only` | int | | +| 14 | `p05_x`, `p50_x`, `p95_x`, `p05_y`, `p50_y`, `p95_y` | int | raw intensity percentiles | +| 15 | `het_rate` | float | autosomal heterozygosity over calls (P5) | +| 16 | `contamination` | float | plan §9.3 (P5; empty if not run) | +| 17 | `_norm_r` | float | one column per assay-control probe present (OUT-062) | +| 18 | `functional_status`, `control_status`, `control_flag_codes`, `flags` | str | QC-030/031/033 (P5; empty before P5) | +| 19… | sheet columns | str | verbatim, sheet order (IN-013) | + + (src: B-9, A-G14, A-G15, E-11, C-G11, plan §13) + +- **OUT-061** `§13 · P4` **Failed samples [EMU].** A sample that fails IDAT → GTC still gets a row: + - `status=FAILED` and `fail_reason` set; + - `autosomal_call_rate = call_rate = 0`; + - `logr_dev`, `_norm_r`, `gc10` and `gc50` are NaN (`NaN` in TSV, `null` in JSON); + - sex is `U`; + - its sheet columns are still passed through. + (src: E-2, B-9) + +- **OUT-062** `§9.1, §13 · P4` **Assay-control locus metric.** + - For each probe named in `--assay-control-probes` (default `TGA_Ctrl_5716`, the targeted-amplification control + of ePGx arrays) that exists in the manifest, `_norm_r` is the normalized R `xn + yn` of that locus (plan + §7.3–§7.4) [INF, OQ-032]. + - The column is omitted when no listed probe exists. + - Values < 1.0 add the flag `assay_control_fail` (QC-033). + (src: A-G32, B-9, C-G10, E-26) + +- **OUT-063** `§13 · P4` **Precision.** Rates and LRR SD use 8 decimal places (`%.8f`); intensities are integers. + NaN is written as `NaN` in TSV. (src: B §2.8) + +- **OUT-064** `§13 · P4` **JSON form.** `sample_summary.json` is an array of flat objects: same keys and order as the + TSV, numbers as JSON numbers, NaN as `null`, sheet values as strings. (src: B-9) + +- **OUT-065** `§13 · P4` **CSV form.** `--summary-format csv|both` writes `sample_summary.csv` with the same columns, + for spreadsheet users. (src: A-G14) + +- **OUT-066** `§13 · P4` **Compatibility column names.** `--compat-column-names` renames the first columns to + reference-style headers: + - `Sample ID`, `Sample Name`, `Sample Folder`; + - `Autosomal Call Rate`, `Call Rate`, `Log R Ratio Std Dev`; + - `Sex Estimate`; + - ` Norm R`. + + It also writes `gt_sample_summary.csv` as an alias file. Our own tools read either form. (src: A-G14, C-G1) + +### 4.6 Other genotyping outputs + +- **OUT-070** `§9.1, §13 · P5` **Control tables.** `simurg genotype` writes `controls_raw.tsv` and + `controls_qc.tsv` (QC-009, QC-011) into the batch folder by default. The genotype output folder is therefore a + complete QC-report dataset (QC-040). (src: A-G14, C-G1) + +- **OUT-071** `§6.7 · P6` **Final report export.** `simurg export finalreport` writes a long-format TSV with one row + per locus × sample. Columns: + - `SNP Name`, `SNP`, `Sample ID`; + - `Allele1 - Top`, `Allele2 - Top`, `Allele1 - Forward`, `Allele2 - Forward`, `Allele1 - Plus`, `Allele2 - Plus`; + - `GC Score`, `GT Score`, `Log R Ratio`, `B Allele Freq`, `Chr`, `Position`. + + Allele rules: + - Top alleles come from the TOP-strand base calls. + - MNV probes are never reverse-complemented (EDGE-016). + - For intensity-only loci, BAF is NaN and LRR uses the cluster-mean expected R (plan §7.6). + + `--matrix` gives a matrix layout. It warns (`W_LARGE_OUTPUT`) when loci × samples > 1e8, with a size estimate. + (src: B-10, C-G23, E-16) + +- **OUT-072** `§7.9 · P6` **Locus summary.** `locus_summary.tsv` has these columns, in order: + - `locus_name`, `locus_id`; + - `n_nocalls`, `n_calls` (both counted over **samples**), `call_freq`; + - `aa_freq`, `ab_freq`, `bb_freq`, `minor_freq`; + - `gentrain_score`, `gc50`, `gc10`; + - `het_excess`, `chi_test_p100`, `cluster_sep`; + - `aa_t_mean`, `aa_t_sd`, `ab_t_mean`, `ab_t_sd`, `bb_t_mean`, `bb_t_sd`; + - `aa_r_mean`, `aa_r_sd`, `ab_r_mean`, `ab_r_sd`, `bb_r_mean`, `bb_r_sd`; + - `plus_minus_strand` (`+`, `-`, `U`). + + `het_excess` follows plan §7.9, the k-regularised legacy form. The simpler form `(f_ab − 2pq)/(2pq + ε)` is written + as `het_excess_simple`, and OQ-030 records the discrepancy between the two. (src: B-11, C-G23) + +- **OUT-073** `§15 · P1` **Run manifest.** `run_manifest.json` follows plan §15, plus: + - `summary` (CLI-009); + - `effective_options` with sources (CLI-005); + - `sheet_header` (IN-008); + - `skipped_loci_by_code`; + - `failed_samples[]`, each with a code; + - `schema_versions` of every output. + + (src: E-47, plan) + +- **OUT-074** `§8.2, §15 · P6` **Audit tables.** `audit/*.parquet` holds: + - one row per harmonisation decision (plan §8.2); + - one row per merge decision, with the probes, calls, candidate count and outcome code; + - one row per skip (OUT-019); + - one row per indel flip (OUT-020). + + (src: B-3, B-5) + +- **OUT-075** `§15 · P6` **Schema discipline [DIFF].** + - Every JSON and TSV output has a versioned schema (`schema` field in JSON; `#simurg_schema=` comment line at the + top of TSVs, which `--no-comment-lines` suppresses). JSON-Schema files are kept under `schemas/`. + - CI validates the golden outputs against them. + - Numbers are numbers; missing values are `null` in JSON and empty or `NaN` in TSV, as documented per column. + - Keys contain no punctuation other than `_`. + - A schema change bumps its version and is logged in `CHANGELOG.md` under "Output schema changes". + (src: E-48, B-18, B-35) + +- **OUT-076** `§15 · P6` **Deterministic ordering.** Every output is sorted by documented keys: + - samples by input order (sheet order, or sorted `sample_id`); + - loci by genome order; + - genes by name; + - solutions by rank, then lexical tie-breakers. + + No output depends on hash-map iteration order, directory listing order, the OS or the thread count. Golden tests + run on x86-64 and aarch64 (plan U11). (src: E-27, E-5) + +- **OUT-077** `§15 · P6` **Default-change changelog.** Any change to a default value of any option or threshold is + listed in `CHANGELOG.md` under "Default changes", with old → new. The reference changed CNV minimum sizes across + minor versions without a machine-readable record. (src: E-47) + +--- + +## 5. QC + +### 5.1 Control-probe metrics + +- **QC-001** `§9.1 · P5` **Control definitions.** + - Controls are read from the BPM control config (`addr[:addr…],Category,Color,Name`) or from the CSV `[Controls]` + section (`Address,Type,Color_Channel,Name`). + - When both manifests are given, the BPM wins, and differences produce one warning. + - Each control address gives a raw `green = Grn.Mean[addr]` and `red = Red.Mean[addr]`, using the raw IDAT means + with no normalization. + (src: A-G27, plan §9.1) + +- **QC-002** `§9.1 · P5` **Canonical control names.** + - Names are normalised before matching: case-fold, collapse whitespace, remove spaces inside parentheses, and treat + `Bgnd` and `Bkg` as the same token. + - They are then mapped to canonical IDs through `simurg/qc/control_names.yaml`. Examples: `Extension (A)` → + `ext_A`; `Biotin (High)` → `biotin_high`; `DNP (Bgnd)` → `dnp_bkg`; `NP (A)` → `np_A`; `String (PM)` → + `string_pm`; `NSB (Bgnd) Red` → `nsb_bkg`; `Hyb (High)` → `hyb_high`. + - Methylation control names (`BS Conversion I C1`, `GT Mismatch 1 (PM)`, …) are mapped the same way. + - When several beads share one canonical ID and a formula uses that probe as a single value, the **mean** is used + [OURS]. Formulas that say min or max take the min or max over all matching beads. + (src: C-G4) + +- **QC-003** `§9.1 · P5` **Backgrounds and offset.** + - `bkg_grn` = the maximum **green** intensity over `Extension (A)` and `Extension (T)`. + - `bkg_red` = the maximum **red** intensity over `Extension (C)` and `Extension (G)`. + - Offset `x = 3000`, configurable with `--bg-offset` or YAML `bgOffset`, value ≥ 0. + + (src: B-13, E-44) + +- **QC-004** `§9.1 · P5, v2` **Metric catalogue.** There are 24 metrics. Table columns: + - **G** = computed for genotyping arrays; **M** = computed for methylation arrays. + - **YAML key** is the threshold key used in QC-020. + - Every metric uses direction `below`: a value < threshold is flagged. + +| # | Column (`controls_qc.tsv`) | YAML key | Formula | G | M | +|---|---|---|---|---|---| +| 1 | `restoration` | `restoration` | Restoration green / (bkg_grn + x) | ✓ | ✓ | +| 2 | `staining_green` | `stainingGreen` | Biotin (High) green / Biotin (Bkg) green | ✓ | ✓ | +| 3 | `staining_red` | `stainingRed` | DNP (High) red / DNP (Bkg) red | ✓ | ✓ | +| 4 | `extension_green` | `extensionGreen` | min(Ext C, Ext G) green / max(Ext A, Ext T) green | ✓ | ✓ | +| 5 | `extension_red` | `extensionRed` | min(Ext A, Ext T) red / max(Ext C, Ext G) red | ✓ | ✓ | +| 6 | `hyb_high_med` | `hybGreenHighMed` | Hyb (High) green / Hyb (Medium) green | ✓ | ✓ | +| 7 | `hyb_med_low` | `hybGreenMediumLow` | Hyb (Medium) green / Hyb (Low) green | ✓ | ✓ | +| 8 | `target_removal_1` | `targetRemovalI` | (bkg_grn + x) / Target Removal (1) green | ✓ | ✓ | +| 9 | `target_removal_2` | `targetRemovalII` | (bkg_grn + x) / Target Removal 2 green | – | ✓ | +| 10 | `stringency` | `stringency` | String (PM) red / String (MM) red (channel configurable, OQ-009) | ✓ | – | +| 11 | `nsb_green` | `nonSpecificBindingGreen` | (bkg_grn + x) / max NSB (Bkg) green | ✓ | – | +| 12 | `nsb_red` | `nonSpecificBindingRed` | (bkg_red + x) / max NSB (Bkg) red | ✓ | – | +| 13 | `np_green` | `npGreen` | min(NP C, NP G) green / max(NP A, NP T) green | ✓ | ✓ | +| 14 | `np_red` | `npRed` | min(NP A, NP T) red / max(NP C, NP G) red | ✓ | ✓ | +| 15 | `specificity_1_green` | `specificityIGreen` | min(GT Mismatch 1–3 PM) green / max(GT Mismatch 1–3 MM) green | – | ✓ | +| 16 | `specificity_1_red` | `specificityIRed` | min(GT Mismatch 4–6 PM) red / max(GT Mismatch 4–6 MM) red | – | ✓ | +| 17 | `specificity_2` | `specificityII` | min(Specificity 1–3 red) / max(Specificity 1–3 green) | – | ✓ | +| 18 | `specificity_2_bg` | `specificityIIBg` | (bkg_grn + x) / max(Specificity 1–3 green) (OQ-012) | – | ✓ | +| 19 | `bs_conv_1_green` | `bisulfiteConversionIGreen` | min(BS Conv I C1, C2) green / max(BS Conv I U1, U2) green | – | ✓ | +| 20 | `bs_conv_1_green_bg` | `bisulfiteConversionIGreenBg` | (bkg_grn + x) / max(BS Conv I U1, U2) green | – | ✓ | +| 21 | `bs_conv_1_red` | `bisulfiteConversionIRed` | min(BS Conv I C3–C5) red / max(BS Conv I U3–U5) red | – | ✓ | +| 22 | `bs_conv_1_red_bg` | `bisulfiteConversionIRedBg` | (bkg_red + x) / max(BS Conv I U3–U5) red (OQ-012) | – | ✓ | +| 23 | `bs_conv_2` | `bisulfiteConversionII` | min(BS Conv II 1–4 red) / max(BS Conv II 1–4 green) | – | ✓ | +| 24 | `bs_conv_2_bg` | `bisulfiteConversionIIBg` | (bkg_grn + x) / max(BS Conv II 1–4 green) | – | ✓ | + + Notes: + - This corrects plan §9.1, where NSB red must use `bkg_red`. + - Restoration and Target Removal II are new relative to the plan. + - Genotyping has 13 control metrics plus the assay-control metric (`assayControl`, OUT-062). Methylation has 21 + (MET-006). + (src: B-13, D-25, C-G5, E-44) + +- **QC-005** `§9.1 · P5` **Genotyping metric set** = rows 1–8 and 10–14 of QC-004, plus `assay_control_norm_r`, which + is copied from the sample summary when present. (src: A-G32, D-25) + +- **QC-006** `§9.1 · v2` **Methylation metric set** = the 21 rows with M ✓. The same computation code is shared with + MET-006. (src: C-G26, D-21) + +- **QC-007** `§9.1 · P5` **Missing controls.** + - Columns that do not apply to the array type are **omitted** from `controls_qc.tsv`. + - When a control a metric needs is absent from the manifest, the metric is `NaN` and `W_CTRL_MISSING` is raised + once per batch. The warning lists the missing control types or names, for example on custom chips. + - These metrics are marked *not evaluable*. They are excluded from pass/fail and listed in `na_metrics` (QC-031). + (src: B-31, D-24) + +- **QC-008** `§9.1 · P5` **Degenerate values.** + - A zero denominator gives `NaN` and `W_CTRL_ZERO_DIVISOR`. Unlike QC-007, a NaN caused by the *signal* (the probe + exists) counts as out of range, so the sample is flagged with the metric's code. + - Negative intensities cannot occur (uint16 input). + (src: plan) [OURS] + +- **QC-009** `§9.1 · P5` **`controls_raw.tsv`** is long format, with one row per control bead per sample: + - columns: `sample_id, category, bead_address, control_name, color_label, green_intensity, red_intensity`; + - row order: sample order, then manifest control order. + + `--compat-column-names` writes `controls.raw_metrics.csv` instead, with columns `SampleId, Category, Beadtype, + Control, Color, GreenIntensity, RedIntensity`. All controls are written, including NEGATIVE and NORM. The exclusions + of QC-010 apply only to the report. (src: B-12, C-G1) + +- **QC-010** `§9.5 · P5` **Report exclusion list.** Raw-probe views and the QC table's raw-intensity columns leave + out: + - the categories NEGATIVE and NORM; + - the methylation-only `NP (G) 1`–`NP (G) 5`; + - `Biotin (5K)` and `DNP (20K)`. + + Derived metrics are still computed from every probe they need. The list is configurable in + `control_names.yaml`. (src: C-G4) + +- **QC-011** `§9.1 · P5` **`controls_qc.tsv`** has one row per sample. Columns, in order: + - `sample_id`, `array_type`; + - `imaging_date` (ISO-8601), `sample_plate`, `sample_well`, `scanner_id`, `scanner_version`: taken from the IDAT + (IN-024), with plate and well overridden by the sheet; + - `bg_offset`; + - the applicable metric columns, in QC-004 order. + + `--compat-column-names` writes `controls.qc_metrics.csv` with `SampleId, ImagingDate, SamplePlate, SampleWell, + ScannerId, ScannerVersion, RestorationQC, StainingGreenQC, …`. Its date format is `M/D/YYYY h:mm:ss AM/PM`. + (src: B-12, C-G1, C-G3) + +- **QC-012** `§9, §14 · P5` **Control-only QC path.** `simurg qc controls` produces QC-009 and QC-011 from IDATs plus + a manifest. It needs no cluster file and does no genotyping, and it works for both genotyping and methylation + arrays. The resulting folder is a valid QC-report dataset with functional QC = `NA`. (src: A-G27, C-G2) + +- **QC-013** `§9, §13 · P5` **Genotype emits controls.** See OUT-070. (src: A-G14) + +- **QC-014** `§14 · P5` **Adding controls to an existing genotype folder.** `qc controls --out ` MAY write + into a folder that already contains a `sample_summary`, for example from an older SimurgArray version or an external + converter, without `--force`, as long as the control tables themselves do not exist yet. The sample IDs must match + the summary; unmatched IDs produce a warning. This mirrors the reference's legacy path. (src: C §1.2) + +### 5.2 Threshold configuration + +- **QC-020** `§9 · P5` **Threshold YAML schema.** Two top-level spellings are accepted: + - `qc:` → `report:` + - `QC:` → `Report:`, for files exported by other tools. + + ```yaml + qc: + profile: genotyping # optional; see QC-024 + report: + callRate: 0.98 # number enables; null (or empty) disables + logRDev: 0.20 + stainingGreen: 5 + stringency: {threshold: 2, direction: below} # mapping form allows direction + restoration: null + assayControl: 1 # alias: tgaControl + bgOffset: 3000 + ``` + - Keys: `callRate`, `logRDev`, `assayControl`, `bgOffset`, and every YAML key in QC-004. + - Aliases: `tgaControl` = `assayControl`; the plan's snake_case `call_rate_min` = `callRate` and `logr_dev_max` = + `logRDev`. + - `direction` is `below` (flag if value < threshold) or `above` (flag if value > threshold). Defaults are `above` for + `logRDev` and `below` for every other key. + - A value exactly equal to the threshold passes. + (src: A-G30, C-G5, E-44) + +- **QC-021** `§9 · P5` **Validation.** Each of these errors exits with code 2: + - the file must exist (`E_QC_CONFIG_MISSING`: "QC config file was provided but does not exist: "); + - the extension must be `.yaml` or `.yml` (`E_QC_CONFIG_EXT`); + - it must be valid YAML (`E_QC_CONFIG_PARSE`); + - the wrapper `qc`/`QC` → `report`/`Report` must be present (`E_QC_CONFIG_STRUCTURE`); + - only known keys are allowed (`E_QC_THRESHOLD_KEY`, which lists the valid keys); + - values must be numeric or null (`E_QC_THRESHOLD_VALUE`); + - `callRate` must be in [0, 1] and every other threshold ≥ 0 (`E_QC_THRESHOLD_RANGE`). + + The same checks apply to `--threshold KEY=VALUE` and to YAML uploaded into the HTML report (QC-055). (src: C-G6, + C-G21) + +- **QC-022** `§9 · P5` **Disabled checks.** A `null` or empty value disables a check. Disabled checks are excluded + from both status axes, from threshold lines, from heatmap binary colouring and from the table's conditional + formatting. (src: A-G30, C 1.8) + +- **QC-023** `§9 · P5` **Precedence and built-ins.** + - Order, highest first: CLI (`--threshold-call-rate`, `--threshold-lrr-sd`, `--threshold KEY=VALUE`), then the + `--config` YAML, then `--profile` (or the YAML's `profile:`), then built-in. + - Built-in (no profile) [EMU]: `callRate = 0.98` and `logRDev = 0.20` are enabled. **Every other threshold is + unset.** + - Example: the YAML sets `callRate: 0.90` and the CLI sets 0.95 → 0.95 is used, with source `cli`. + - Each effective threshold is recorded with its source (CLI-005, QC-045). + (src: A-G30, C-G6) + +- **QC-024** `§9 · P5, v2` **Shipped profiles** (`simurg/qc/profiles/*.yaml`) are *starting points for review*, not + truths. The documentation says they came from the centre and spread of background distributions in a small set of + good and failed datasets, and that they should be recalibrated with QC-026. Values where the reference's own + documents disagree are listed in OQ-007/OQ-008. + +| Key | `genotyping` (LCG/HTS) | `genotyping_ex` (EX; provisional, OQ-046) | `genotyping_example` (alternative values) | `methylation_epic` | `methylation_msa` | +|---|---|---|---|---|---| +| callRate / logRDev | 0.98 / 0.20 | 0.98 / 0.20 | 0.98 / 0.15 | – | – | +| stainingGreen / Red | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | +| extensionGreen / Red | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | +| hybGreenHighMed / MediumLow | 1 / 1 | 1 / 1 | 1.25 / 1.25 | 1 / 1 | 1 / 1 | +| targetRemovalI / II | 1 / – | 1 / – | 5 / – | 1 / 1 | 1 / 1 | +| restoration | null (1 with `+ffpe`) | null | null | 0 (1 with `+ffpe`) | 0 (1 with `+ffpe`) | +| stringency | 2 | 2 | 1.5 | – | – | +| nonSpecificBindingGreen / Red | 1 / 1 | 1 / 1 | 5 / 5 | – | – | +| npGreen / npRed | 5 / 5 | 5 / 5 | 5 / 5 | 5 / 5 | 2.5 / 3 | +| assayControl | 1 | 1 | 1 | – | – | +| bisulfiteConversionIGreen / Red | – | – | – | 1 / 1 | 1 / 1 | +| bisulfiteConversionIGreenBg / RedBg | – | – | – | 1 / 1 | 0.5 / 0.5 | +| bisulfiteConversionII / IIBg | – | – | – | 1 / 1 | 0.5 / 0.5 | +| specificityIGreen / Red / II / IIBg | – | – | – | 1 / 1 / 1 / 1 | 1 / 1 / 1 / 1 | +| bgOffset | 3000 | 3000 | 3000 | 3000 | 3000 | + + - The suffix `+ffpe` (e.g. `--profile methylation_epic+ffpe`) sets `restoration: 1`. + - Methylation profiles also carry `pvalThreshold: 0.05` (MET-011). + (src: C-G7, B-14, D-21, A-G30) + +- **QC-025** `§9 · P5` **Profile auto-selection.** `--profile auto` picks the profile from the array type (dataset + detection, QC-043) and the manifest `Assay Format`: `EX` → `genotyping_ex`, and everything else → `genotyping`. For + methylation it goes by manifest name: EPIC → `methylation_epic`, MSA → `methylation_msa`. The report shows which + profile was used for each dataset. (src: B-14, C-G7) + +- **QC-026** `§9 · P5` **Data-driven threshold suggestion.** `simurg qc suggest-thresholds` reads one or more + datasets. For every enabled or known metric it computes a robust centre `m` (the median) and a spread + `s = 1.4826·MAD` over samples, on the log scale for ratio metrics. + - It suggests `exp(m − k·s)` for `below` keys and `exp(m + k·s)` for `above` keys, with k = 4 by default. + - It needs ≥ 30 samples per metric; otherwise the metric is left unset, with a note. + - Output: a YAML file (QC-020 schema, with comments showing m, s, n and the % flagged at both the current and the + suggested value) and a TSV of the same numbers. + (src: C-G8) + +### 5.3 QC status semantics + +- **QC-030** `§9.2 · P5` **Functional status.** Values: `PASS`, `FAIL` or `NA`. + - `PASS` iff every enabled functional check passes: `autosomal_call_rate ≥ callRate` **and** `logr_dev ≤ logRDev`. + - `FAIL` if any enabled check fails. Failed samples (OUT-061) are `FAIL`. + - `NA` when no genotyping summary exists: control-only datasets and methylation. + - Overall call rate is reported but never drives the status. + (src: C-G9, C-G11) + +- **QC-031** `§9.2 · P5` **Control status.** Values: `CLEAR` or `FLAGGED`. + - `FLAGGED` iff any enabled control metric is out of range, including `assayControl`. + - `control_flag_codes` lists the failing YAML keys, `;`-joined in QC-004 order (e.g. `npRed;stringency`). + - `na_metrics` lists metrics that could not be evaluated (QC-007). + - The two status axes are independent: a sample can be `PASS` and `FLAGGED`. The documentation explains how to read + the combination: biological or sample failures versus technical assay steps. + (src: C-G9, C 1.8) + +- **QC-032** `§9.2 · P5` **Methylation pass.** `pass_qc = 1` iff every enabled methylation control metric passes; + `fail_codes` lists the failures (MET-008). Probe detection rate never affects it unless `--min-prop-passing` is + set. (src: D-21, D-22) + +- **QC-033** `§9.2 · P5` **Sample QC flags** (`flags` column, `;`-joined, informational): + - `low_call_rate`, `high_lrr_sd`: mirror the functional checks; + - `sex_mismatch`; + - `het_outlier` (|z| > 3 within the batch); + - `contaminated` (plan §9.3 estimate > 0.02 [OURS]); + - `assay_control_fail` (OUT-062); + - `duplicate_of:`, `unexpected_relatedness` (plan §9.4). + + (src: plan §9.2) + +- **QC-034** `§9.2 · P5` **Assay-control metric in QC.** `assayControl` (default threshold 1 in the genotyping + profiles) takes part in control status, in the trend statistics (QC-054) and in the PGx review gate (QC-035). + (src: C-G10, A-G32) + +- **QC-035** `§9.2, §11 · P5` **Recommended downstream gates.** These are documented in the recipes (CLI-040) and + applied as `pgx_review` and `cn_training_exclude` flags: + - PGx review: `logr_dev > 0.2`, `call_rate < 0.99`, or `assay_control_norm_r < 1.0`. + - CN training: the same three conditions (PCN-005). + - Cyto: `logr_dev > 0.2` → SampleQuality; `> 0.3` → whole-chromosome-only mode (CYTO-014). + (src: A-G65, A-G34) + +### 5.4 QC report — inputs, outputs, datasets + +- **QC-040** `§9.5 · P5` **Input contract.** The report reads **only** dataset folders that contain: + - `controls_raw.tsv` and `controls_qc.tsv` (or their compat CSV names): required; + - `sample_summary.tsv` (or `.csv`, or the compat `gt_sample_summary.csv`): optional but recommended, because it + enables functional QC, call-rate and LRR SD views, heatmaps and metadata. + + It never touches raw data, so reports can be regenerated or merged later. Joins use `sample_id`. IDs present in one + file but not the other → `W_QC_UNMATCHED_SAMPLES`, listing up to 20 of them. Those samples appear with the fields + that are available. (src: C-G1, A-G28, C 1.13) + +- **QC-041** `§9.5 · P5` **`--data` forms.** + 1. A folder that contains the required files. + 2. A comma list of such folders. + 3. A parent folder, searched recursively for every subfolder containing both control files. Discovered folders are + sorted by path. + + Error cases: + - A missing path → `E_QC_NO_DATASETS`: "Input data folder does not exist: ". + - No valid folder → "No valid dataset folders detected. Each folder must include at least: controls_raw.tsv, + controls_qc.tsv". + - A listed folder lacking files → `E_QC_DATASET_INCOMPLETE`: "Dataset