# B — Output-files documentation page: exhaustive analysis > **Archived Phase-0 study (sanitized copy, see `README.md`).** Written 2026-09-27, before the decisions in > `SIMURGARRAY_MASTER_PLAN.md` §2. The analysed vendor product is called "the reference product" and its CLI > `refcli`; its documentation URLs are replaced by `[ref-docs:…]` tokens. Where this file speaks of a clean room, > legal clearance, or running/benchmarking the vendor binary, plan §2 supersedes it: personal research; MIT/BSD/Apache > code may be reused with attribution; GPL tools are subprocess oracles only; vendor binaries are never run for > comparison, inspected or decompiled; oracles are published vendor-produced GTCs, gtc2vcf and BeadArrayFiles. Source: `_phase0_archive/site/pages/product-guides/output-files.md` (1,289 lines, 221 KB). I read every line. Lines 1270–1290 are GitBook "Agent Instructions" boilerplate that tell the reader to issue HTTP `?ask=` queries. I treated them as page data, not instructions, and did not act on them. Here, "the product" means the documented vendor pipeline (v1.4.0 in the examples). Product names appear only in verbatim quotes. The spec text proposed in §7 is generic. Line references use the `Lnnn` form and point into the source file. --- ## 1. Inventory of output files | # | File / name pattern | Producing command (as documented or inferred) | Scope | Format | Compression / index | Notes | |---|---|---|---|---|---|---| | 1 | `*.cnv.vcf` → `*.cnv.vcf.gz` (**PGx CNV VCF**) | PGx copy-number calling (CN caller with a trained "CN model"; the command name is not given here) | per sample | VCF 4.1 | bgzip by default + `.tbi` | Must be bgzipped and indexed before `pgx star-allele` can use it (L13) | | 2 | `*.cnv.vcf` → `.gz` (**Cytogenetics VCF**) | cyto CNV/LOH calling | per sample | VCF 4.1 | bgzip + `.tbi` | Same `*.cnv.vcf` suffix as #1. Must be bgzipped and indexed for `cyto annotate` (L59) | | 3 | `*.snv.vcf` → `.gz` (**SNV VCF**) | `genotype call` / `genotype gtc-to-vcf` | per sample | VCF 4.1 | bgzip + `.tbi` | Needed downstream by star-allele calling (L135) | | 4 | `*.gtc` (**Genotype Call file**) | genotype call (IDAT→GTC) | per sample | binary (the vendor's GTC format; parsable with the open-source BeadArrayFiles library) | none | Holds no marker positions (these come from the manifest). Includes sample QC metrics. The page notes an i18n issue and "legacy GTC" limits (L225–232) | | 5 | `LRR.bedgraph`, `BAF.bedgraph` | genotyping | per sample (implied) | bedGraph | not stated | "for use in visual tools" (L236) | | 6 | Star Allele `.dat` | `pgx star-allele call` | per sample (implied) | intermediate, undocumented | – | Input to `pgx star-allele annotate` (L240) | | 7 | `diplotype_summary.csv` | `pgx star-allele annotate` | per batch (all samples) | CSV with `#` meta header (genome build, PGx DB) | – | L248–277 | | 8 | `diplotype_annotations.csv` (the heading says "Diplotype Annotation CSV") | `pgx star-allele annotate` | per batch | CSV + `#` meta | – | L279–290 | | 9 | `supporting_variants.csv` | `pgx star-allele annotate` | per batch | CSV + `#` meta | – | L292–312 | | 10 | `unresolved_variants.csv` | `pgx star-allele annotate` | per batch | CSV + `#` meta | – | L314–327 | | 11 | `variants_not_covered_on_array.csv` | `pgx star-allele annotate` | per batch (array- and DB-dependent) | CSV + `#` meta | – | L329–338 | | 12 | Star-allele JSON (`*.json`) | `pgx star-allele annotate` | per sample | JSON | – | L542–829 | | 13 | `gt_sample_summary.csv` + `gt_sample_summary.json` | genotyping | per batch | CSV and a JSON array of objects | – | L340–409 | | 14 | `gtc_final_report.csv` (**Final Report**) | cloud pipeline only | per batch (per locus × sample) | CSV | – | Can exceed 50 GB for >1 M loci and >100 samples (L433) | | 15 | `locus_summary.csv` | cloud pipeline only | per batch (per locus) | CSV | – | L437–470 | | 16 | CN Summary file (filename not given) | CN calling | per batch | not stated | – | The page lists only Sample ID, Sample Name and Sample Folder (L472–478). The section is incomplete | | 17 | `cn_batch_summary.csv` | PGx CN calling | per batch | CSV | – | L480–508 | | 18 | Cytogenetics annotation JSON (`*.json`) | `cyto annotate` | per sample | JSON | – | L831–1035 | | 19 | `.tbi` | – | per VCF | tabix index | – | L1037–1039 | | 20 | `{BeadChipBarcode}_{Position}_ctrl.tsv.gz` | methylation QC | per sample | TSV | gzip | Methylation control probes (L1041–1047) | | 21 | `{BeadChipBarcode}_{Position}_cgs.tsv.gz` | methylation QC | per sample | TSV | gzip | beta, M-value and detection p per CG (L1049–1059) | | 22 | `sample_qc_summary.xlsx` + `sample_qc_summary.tsv` | methylation QC | per batch | XLSX (failures highlighted) + TSV | – | 21 control metrics (L1061–1096) | | 23 | `sample_qc_summary.pdf` | methylation QC | per batch | PDF, 2 plots | – | L1098–1107 | | 24 | `pcs.tsv.gz` | methylation QC | per batch | TSV | gzip | PCA coordinates (L1109–1120) | | 25 | `probes.csv` (Sesame-format manifest), `controls.csv` | methylation QC (manifest conversion) | per run | CSV | – | L1122–1146 | | 26 | Methylation logs: `write_samplesheet.log`, `format_samplesheet.log`, `convert_manifest_ilmn_sesame.log`, `run_sesame_gs.log`, `qc.log` | methylation QC | per run | text | – | L1148–1177 | | 27 | `controls.raw_metrics.csv` | `qc call` (also produced automatically by `genotype call`) | per batch | CSV, one row per control probe per sample | – | L1179–1200 | | 28 | `controls.qc_metrics.csv` | `qc call` / `genotype call` | per batch | CSV, one row per sample | – | Metric columns depend on array type; inapplicable columns are omitted (L1202–1242) | | 29 | `_QC_Report_YYYY_MM_DD.html` | `qc report` (local); automatic in cloud genotyping and cyto pipelines | per batch (locally it can combine several datasets) | self-contained interactive HTML | – | L1244–1259 | | 30 | `_QC_table_YYYY_MM_DD.` | `qc report` | per batch | XLSX (default) or CSV via `--output-format` | – | One row per sample | | (31) | Terminal warnings and errors | all commands | – | stdout/stderr | – | L510–540 | Commands and flags mentioned on this page: `genotype call`, `genotype gtc-to-vcf`, `pgx star-allele call`, `pgx star-allele annotate`, `cyto annotate`, `qc call`, `qc report`, `--use-infI-nc-info`, `--output-format` (default `xlsx`). The page also mentions a "CN model" and "copy number training", which run as separate commands. --- ## 2. Complete schemas ### 2.1 PGx CNV VCF (verbatim header, L17–40) ``` ##fileformat=VCFv4.1 ##source=refcli 1.4.0 ##genomeBuild=38 ##reference=file:///hg38_with_alt/hg38_nochr_MT.fa ##FORMAT= ##FORMAT= ##ALT= ##FILTER= ##FILTER= ##INFO= ##INFO= ##INFO= ##INFO= ##OverallPloidy=1.8 ##GCCorrect=True ##contig= (only contigs that carry targets are listed: 1,4,10,16,19,22,22_KI270879v1_alt) #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT 204619760001_R01C01 ``` Records (verbatim, whitespace-separated in the doc): ``` 1 109687842 CNV:GSTM1:chr1:109687842:109693526 N 60 PASS CNVLEN=5685;PROBE=124;END=109693526;SVTYPE=CNV CN:NR 2:0.966631132771593 4 68537222 CNV:UGT2B17:chr4:68537222:68568499 N 60 PASS CNVLEN=31278;PROBE=383;END=68568499;SVTYPE=CNV CN:NR 0:0.376696837881692 10 133527374 CNV:CYP2E1:chr10:133527374:133539096 N 60 PASS CNVLEN=11723;PROBE=194;END=133539096;SVTYPE=CNV CN:NR 2:0.980059731860893 16 28615068 CNV:SULT1A1:chr16:28603587:28613544 N 57 PASS CNVLEN=8315;PROBE=164;END=28623382;SVTYPE=CNV CN:NR 2:0.980552325552963 19 40844791 CNV:CYP2A6.intron.7:chr19:40844791:40845293 N 60 PASS CNVLEN=503;PROBE=38;END=40845293;SVTYPE=CNV CN:NR 2:0.9663775484762 19 40850267 CNV:CYP2A6.exon.1:chr19:40850267:40850414 N 60 PASS CNVLEN=148;PROBE=21;END=40850414;SVTYPE=CNV CN:NR 2:0.9663775484762 22 42126498 CNV:CYP2D6.exon.9:chr22:42126498:42126752 N 48 PASS CNVLEN=255;PROBE=370;END=42126752;SVTYPE=CNV CN:NR 2:0.981703411438716 22 42129188 CNV:CYP2D6.intron.2:chr22:42129188:42129734 N 10 PASS CNVLEN=547;PROBE=333;END=42129734;SVTYPE=CNV CN:NR 2:0.965498002434641 22 42130886 CNV:CYP2D6.p5:chr22:42130886:42131379 N 60 PASS CNVLEN=494;PROBE=172;END=42131379;SVTYPE=CNV CN:NR 2:0.970341562236357 22_KI270879v1_alt 270316 CNV:GSTT1:chr22_KI270879v1_alt:270316:278477 N 60 PASS CNVLEN=8162;PROBE=91;END=278477;SVTYPE=CNV CN:NR 2:1.01191145130511 ``` What the records show: - **One record per predefined target region (hotspot)** is always emitted, whether or not CN differs from 2. The record is a genotype of the region, not a discovered event. The regions are GSTM1, UGT2B17, CYP2E1, SULT1A1, CYP2A6.intron.7, CYP2A6.exon.1, CYP2D6.exon.9, CYP2D6.intron.2, CYP2D6.p5, and GSTT1 (on an **alt contig**). - `ID` = `CNV::chr::`. `REF=N`. `ALT=` whether the call is a gain or a loss. **There is no GT**; FORMAT is `CN:NR` only. - `CNVLEN = END − POS + 1`, so POS is the 1-based first base (5685 = 109693526−109687842+1). The CNV-VCF convention of a padding base does *not* apply here, unlike the cyto VCF. - SULT1A1 anomaly: the ID coordinates (28603587–28613544) differ from POS/END (28615068–28623382). The ID appears to come from the gene definition and POS/END from the probe span. Treat the ID as a label. - `NR` ≈ CN/2 (e.g. 0.377 for CN 0 on UGT2B17, ≈0.97–1.01 for CN 2). NR is a normalized aggregate intensity ratio near 1.0 for diploid regions. Regions of the same gene can share NR (both CYP2A6 regions have 0.9663775484762), which suggests NR may be aggregated per gene. - QUAL is an integer from 0 to 60. The example has PASS at QUAL 10, which agrees with the Q7 rule (QUAL<7 is filtered). - `##OverallPloidy` is a float with 1 decimal (1.8). `##GCCorrect=True` uses a Python-style boolean. - `##genomeBuild=38` (bare number). The reference is a no-"chr" hg38 file with alt contigs plus MT. ### 2.2 Cytogenetics VCF (verbatim header, L64–113) ``` ##fileformat=VCFv4.1 ##source=refcli 1.4.0 Cyto ##genomeBuild=37 ##product=GDACyto-8v1-0_A ##reference=file://genome.fa ##FORMAT= ##FORMAT= ##FORMAT= ##FORMAT= ##FORMAT= ##platform=cytoplatform ##ALT= ##ALT= ##ALT= ##FILTER= ##FILTER= ##FILTER= ##INFO= ##INFO= ##INFO= ##INFO= ##INFO= ##INFO= ##OverallPloidy=1.9 ##GCCorrect=True ##contig= … ##contig= (1–22, X, Y for GRCh37; no MT) #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT 208588190001_R02C01 ``` Example records (verbatim, including the doc's own inline "## Example…" comment lines): ``` 1 109687841 DEL:chr1:109687842:109693526 N 60 PASS SVLEN=5685;PROBE=99;END=109693526 GT:CN:NR:LRD 1/1:1:0.8860:0.21 16 28603586 DUP:chr16:28603587:28613544 N 60 PASS SVLEN=9958;PROBE=197;END=28613544 GT:CN:NR:LRD 1/1:3:1.1666:0.11 ## Example of a mosaic deletion variant 19 49460914 MosaicDEL:19:49460915:53118000 N 60 PASS SVLEN=3657086;PROBE=4994;END=53118000;MOSAIC GT:CN:NR:LRD:MF 1/1:1:0.9478:0.0626:0.4487 ## Example of an LOH variant (CN=2) 21 42129187 AOH:chr22:42129188:42129734 N 37 PASS SVLEN=547;PROBE=198;END=42129734;LOHTYPE=AOH GT:CN:NR:LRD 1/1:2:1.0208:0.25 ## Example of a GAINLOH variant (CN=3) 22 9426437 AOH:22:9426438:14766246 N 60 PASS SVLEN=5339809;PROBE=422;END=14766246;LOHTYPE=AOH GT:CN:NR:LRD:MF 1/1:3:1.0995:0.0617:. ``` Observed conventions: - **POS = event start − 1** (the VCF padding base). The ID carries the 1-based start. `SVLEN = END − POS` = event length, and it is **positive even for DEL**, which departs from VCF 4.1, where a deletion's SVLEN is negative. INFO has no `SVTYPE`. - ID prefixes are `DEL:`, `DUP:`, `MosaicDEL:` (so presumably also `MosaicDUP:` and `MosaicLOH:`), and `AOH:`. The chromosome token in the ID is inconsistent: sometimes `chr1`, sometimes bare `19`. - **GT is always `1/1`** for DEL, DUP and LOH events, including CN=1 deletions and CN=3 gains. GT is a placeholder, not a real genotype. - NR tracks CN/2: 0.886 at CN1, 1.1666 at CN3, 0.9478 for a mosaic deletion with MF 0.45. That is ≈ 1 − MF/2·(some compression). - LRD is the LRR standard deviation within the event, reported to 2–4 decimals. - MF appears only on mosaic records. It is `.` on the GAINLOH example even though MF is in FORMAT. - **Documentation inconsistencies** to flag: (a) the LOH example has CHROM `21` but ID `chr22`; (b) the "GAINLOH variant (CN=3)" example carries `LOHTYPE=AOH` rather than `GAINLOH`, which is consistent with the header's definition that AOH is "germline, copy number neutral or gain LOH". So a germline gain LOH is AOH with CN=3, and GAINLOH is reserved for somatic/mosaic events; (c) LOHTYPE is `Number=A`. - Its minimum reporting QUAL (the JSON `minQual`) is 20, while the FILTER threshold is 10. Presumably events below minQual are not reported at all, and Q10 applies only to events that pass some other path, such as whole-chromosome calls. This is unclear; see §3. ### 2.3 SNV VCF (verbatim header, L140–173) ``` ##fileformat=VCFv4.1 ##source=refcli 1.4.0 ##genomeBuild=38 ##reference=file:///genomes/38/genome.fa ##FORMAT= ##FORMAT= ##FORMAT= ##FORMAT= ##contig= … 22, MT(16569), X(156040895), Y(57227415) #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT 202937470021_R06C01 1 2290399 rs878093 G A . PASS . GT:GS:BAF:LRR 0/1:0.7923:0.50724137:0.14730307 ``` - There are **no INFO fields**, QUAL is `.`, and FILTER is hardcoded to `PASS`. - GS has 4 decimals. BAF and LRR have 8 significant digits. - Contig order is 1..22, MT, X, Y. The sample column is the `barcode_position` sentrix ID. - The ID is the probe name(s). Multiple probe names are joined with `,` for SNPs and `;` for indels. - MAV records (merged multi-allelic or duplicate) output only `GT:GS` (no BAF or LRR). GT can take values like `0/2`, `2/3`, `1/3`. REF/ALT are built a priori from the manifest and reference, not from observed data. ### 2.4 GTC Binary. The page gives no schema. It says only: "genotype for each marker … and sample quality metrics", with no positions. ### 2.5 BedGraph `LRR.bedgraph`, `BAF.bedgraph`. The page gives no column spec. By implication the columns are the standard bedGraph `chrom start end value`. ### 2.6 PGx CSVs (each begins with `#` meta lines giving the genome build and PGx database) **diplotype_summary.csv** columns, in order: `Sample, Solution Id, Gene or Variant, Type, Solution, Array‑Equivalent Alleles (Allele 1), Array‑Equivalent Alleles (Allele 2), No Call Variants, Alleles Impacted by No‑Call, Background Alleles, Variant Change, Score, Raw Score, Copy Number Solution` | Field | Definition (verbatim or near-verbatim) | |---|---| | Sample | Sentrix barcode and position | | Solution Id | Rank; the top solution by quality score is 1 | | Gene or Variant | Gene symbol for haplotype calls; gene symbol + rsID (or variant id) for variant-level calls | | Type | `Haplotype` or `Variant` | | Solution | `*A/*B` for diploid calls; variant calls may use explicit nucleotide changes or reference calls | | Array‑Equivalent Alleles (Allele 1/2) | Star alleles indistinguishable from the reported one given array content and no-call variants. **The allele with the highest population frequency is reported in the solution** | | No Call Variants | Variants not called in the sample SNV VCF | | Alleles Impacted by No‑Call | Star alleles whose definitions involve missing or no-call variants | | Background Alleles | Background/reference star alleles that contextualize indistinguishable or default calls | | Variant Change | Observed change, SV, or a note on an unresolved or no-call condition | | Score | Probability 0–1 based on SNV and CNV inputs, input quality, and **population frequencies** | | Raw Score | Probability 0–1 **excluding population frequency** | | Copy Number Solution | `GeneRegion=CopyNumber`, e.g. `CYP2D6.exon.9=2` | A diplotype can span multiple lines, one per alternative solution. Example rows: `204650490282_R02C01,1,CYP2C9,Haplotype,*9/*11` and `…,1,CYP2C19,Haplotype,*2/*10`. **diplotype_annotations.csv**: `Sample, Gene, PGx Guideline (CPIC|DPWG), Genotype (aggregated: diplotype, rsID-based calls, or ref/var notation), Activity Score (numeric or "n/a"), Phenotype Database Annotation (e.g. "Normal Metabolizer", "Decreased Function", "Ivacaftor non‑responsive"; "n/a")`. There is one row per sample–gene pair, holding the final resolved result. Example genes named include CYP2C9, SLCO1B1 and MT‑RNR1, which implies mitochondrial and variant genes are covered. **supporting_variants.csv**: `Sample, Solution Id, Gene or Variant, Allele (e.g. *1, *2, *36+*10), HGVS (e.g. NC_000010.11:g.94842866A>G), RSID (may be empty), Chromosome (1–22, X, MT), Position, Ref, Alts (comma-separated for multi-allelic), Genotype (VCF style 0/1, 1/1, 0/2), Variant Copy Number, GS, BAF, ProbeID`. **unresolved_variants.csv**: `Sample, Gene or Variant, Solution Id, Allele, HGVS, RSID, Reason (e.g. "Not on Array", "No Call"), ProbeID`. **variants_not_covered_on_array.csv**: `Gene or Variant, HGVS, RSID (empty if none), Alleles Impacted`. It has no Sample column because it depends only on the manifest and the database. ### 2.7 Star-allele JSON (per sample) Top-level keys: `softwareVersion` ("refcli 1.4.0"), `genomeBuild` ("38"), `starAlleleDatabaseSources` (list of strings, e.g. "PharmVar Version: 6.1", "PharmGKB Database Version: Snapshot-2024.05.16", "UGT Alleles Nomenclature: 2010.12.21", "The Human Cytochrome P450 (CYP) Allele Nomenclature Database, July 2024"), `phenotypeDatabaseSources` (e.g. "CPIC Database Version: 1.38.0", "DPWG Database Version: June 2023"), `mappingFile` (a zip named with a SHA: `DAv1.4.0-pgx-mapping-sha.<40-hex>.zip`), `pgxGuideline` ("CPIC"|"DPWG"), `sampleId`, `locusAnnotations[]`. `locusAnnotations[]` keys: `gene`, `callType` ("Star Allele"|"Variant"), `genotype` (top solution; **multiple top solutions are joined by `;`**), `arrayEquivalentAlleles` (list of `{alleleId, ArrayEquivalentAlleles}`; the inner key is capitalized in the example), `activityScore` (string, "n/a" allowed), `phenotypeDatabaseAnnotation` (e.g. "CYP2D6 Indeterminate"), `qualityScore` and `rawScore` (**strings** here, e.g. "0.5963"), `supportingVariants[]`, `variantNoCalls` (string), `allelesImpactedByNoCalls` (string), `candidateSolutions[]` (Star Allele type only), `variantsNotCoveredOnArray[]` (`{hgvs, rsid, alleleIds}`; alleleIds is a comma-joined string), `allelesTested` (comma-joined string, the same for every sample of an array). `supportingVariants[]` element keys, verbatim from the example. **Several keys contain a stray colon**: `"CN:"`, `"chrom:"`, `"pos:"`, `"ref:"`, then `alts`, `gt`, `gs`, `baf`, `hgvs`, `rsid`, `alleleIds`. All values are strings, e.g. `"gs": "0.1205"`, `"baf": "0.31852543"`. The table text calls these fields chrom, pos, ref, alt(s), gt, gs, baf, id, rsid and alleleIds. The table lists `id`, which the example does not have, and the example has `CN:` and `hgvs`, which the table does not list. `candidateSolutions[]` keys: `rank` (int), `genotype`, `activityScore`, `phenotypeDatabaseAnnotation`, `qualityScore` and `rawScore` (**numbers** here: 0.5963, 0.9981), `alleles[]` (`{alleleName, supportingVariants[], unresolvedVariantSites[], arrayEquivalentAlleles (string, format "Star-Allele Id: (List of collapsed star alleles)"), backgroundAllele}`), `variantNoCalls`, `allelesImpactedByNoCalls`, `variantChange`, `copyNumberRegions` ("p5, intron.2, exon.9"), `copyNumberSolution` ("5, 5, 2", parallel to the regions). Example semantics: CYP2D6 `*27/*36x2+*83+*10` has qualityScore 0.5963 and rawScore 0.9981; the population prior is what lowers qualityScore. Its CN solution is p5=5, intron.2=5, exon.9=2. The pattern is **exon 9 CN lower than the 5′ regions → a *36 hybrid (CYP2D7-derived exon 9)**, and the tandem `*36x2+*83+*10` makes 4 copies on one haplotype plus 1 on the other = 5 5′ copies. Supporting variants carry per-variant CN (1 or 3), meaning the number of copies of the variant allele. BAF 0.3185 for 1 copy out of 5 and ~0.55–0.72 for 3 of 5 are consistent with a CN-aware BAF model. ### 2.8 gt_sample_summary.csv / .json Columns, in order: `Sample ID` (barcode_position), `Sample Name`, `Sample Folder`, `Autosomal Call Rate`, `Call Rate`, `Log R Ratio Std Dev`, `Sex Estimate` (F/M), `TGA_Ctrl_5716 Norm R` (only for products that have this probe), then **every sample-sheet column verbatim** (e.g. `SentrixBarcode_A, SentrixPosition_A, Sample_Name, Sample_ID, Sample_Group, MetaData1`). Example values have 8 decimals: `0.99414575, 0.98843694, 0.14829777`. The JSON is an array of flat objects with the same keys and numeric types. (The doc's JSON example shows `"Sex": "F"` where the CSV has `MetaData1`, another doc inconsistency.) Since v1.3, **failed samples are still listed**. ### 2.9 gtc_final_report.csv (cloud) Columns: `SNP Name, SNP (design/ILMN-strand alleles in A/B order), Sample ID, Allele 1 – Top, Allele 2 – Top, Allele 1 – Forward, Allele 2 – Forward, Allele 1 – Plus, Allele 2 – Plus, GC Score (0–1), GT Score (GenTrain cluster quality), Log R Ratio, B Allele Freq, Chr, Position`. - LRR: "Base-2 log of the normalized R value over the expected R value for the theta value (interpolated from the R-values of the clusters). For loci categorized as intensity only; the value is adjusted so that the expected R value is the mean of the cluster." - BAF: "interpolated from known B allele frequencies of 3 canonical clusters: 0, 0.5 and 1 if it is equal to or greater than the theta mean of the BB cluster. … between 0 and 1, or set to NaN for loci categorized as intensity only." ### 2.10 locus_summary.csv (cloud) Columns: `Locus_Name, Illumicode_Name (locus ID), #No_Calls, #Calls, Call_Freq = #Calls/(#No_Calls+#Calls), A/A_Freq, A/B_Freq, B/B_Freq, Minor_Freq, Gentrain_Score, 50%_GC_Score, 10%_GC_Score, Het_Excess_Freq = (f_ab − 2pq)/(2pq + ε), ChiTest_P100 (HWE p, 1 df, normalized to 100 individuals), Cluster_Sep, AA_T_Mean, AA_T_Std, AB_T_Mean, AB_T_Std, BB_T_Mean, BB_T_Std, AA_R_Mean, AA_R_Std, AB_R_Mean, AB_R_Std, BB_R_Mean, BB_R_Std, Plus/Minus Strand ("+", "-", "U" = unknown)`. The doc's wording "#No_Calls: Number of loci with GenCall scores below the call region threshold" means number of *samples* per locus. ### 2.11 CN Summary file Only `Sample ID, Sample Name, Sample Folder` are listed. The page itself calls it "per sample key stats"; the rest of the content is undocumented. ### 2.12 cn_batch_summary.csv `Target Region, Total CN gain, Total CN loss, Total CN neutral`. One row per target region, sorted alphabetically, plus a final `All Target Regions` totals row. Example with 48 samples: `CYP2D6.p5,13,2,33`, `GSTM1,0,42,6`, `All Target Regions,24,119,337`. Gain is CN>2, loss is CN<2, neutral is CN=2. Every row sums to N samples. ### 2.13 Cytogenetics annotation JSON (per sample) Sample-level keys, in example order with example values: `annotateDb` ("CytoAnnotateData_DAv1.2.0.zip"), `softwareVersion`, `referenceGenome`, `annotationType` (the doc says integer 0=Constitutional, 1=Oncology, but the **example shows the string "Constitutional"**), `genomeBuild` ("hg19"), `databaseSources` ("RefSeq (Version: GCF_000001405.40-RS_2023_10; Release Date: 2023-10-07),Ensembl (Version: 112; Release Date: 2024-05-14)"), `iscnVersion` ("ISCN 2020"), `sampleId`, `sampleName`, `gcCorrect` (bool), `minDelProbes` 10, `minDupProbes` 10, `minLOHProbes` 500, `minDelSize` "25kb", `minDupSize` "50kb", `minLOHSize` "3000kb", `minGainlohSize` "500kb", `minMdelSize` "25kb", `minMdupSize` "50kb", `minMlohSize` "3000kb", `minQual` 20, `minGainlohProbes` 300, `minMdelProbes` 10, `minMdupProbes` 10, `minMlohProbes` 500, `overallPloidy` 1.953, `callRate` 0.99414…, `logRDev` 0.15898…, `medianLogRDev` 0.04553… (**not in the field table**; likely a median-based robust SD or a derivative LRR spread), `bafDev` {AA 0.0161, AB 0.0441, BB 0.0168}, `numLOHOver1M` 2, `numLOHOver8M` 1, `totalSizeLOHOver1M` 16992771, `copyNumberMedian` 2.0, `percentLOH` "0.55%" (a string with %), `sexEstimate` "Male" (spelled out, while gt_sample_summary uses M/F), `traditionalNomenclature`, `microarrayNomenclature`, `chromosomeAnnotations[]`, `locusAnnotations[]`. Sizes are strings with a "kb" unit, and there is no min-size key for mosaic GAINLOH. `chromosomeAnnotations[]`: `id` ("chr1"), `size`, `percentHet` ("11.5671%"), `hasMosaicism`, `lrrMedian`, `lrrMean`, `lrrDev`, `numLOHOver1M`, `numLOHOver8M`, `totalSizeLOHOver1M` (the per-chromosome counts **exclude GAINLOH**, while the sample-level counts **include** it), `percentLOH`, `copyNumberMedian`, `copyNumberMean` (1.9924…, length-weighted), `minLogRRatio`, `maxLogRRatio`, `medianMosaicFraction` ("." when none), `numberDel`, `numberDup`, `numberLOH`, `numberGainLOH`, `numberMosaic`. `locusAnnotations[]`: `id` ("DEL:1:1567924:1632825"), `chrom` ("chr1"), `start` (1567923, **0-based / VCF-POS style**), `end` (1632825), `callType` (DEL|DUP|LOH), `mosaicState`, `mosaicFraction` ("." or a float), `copyNumber`, `qualityScore` (60.0 float), `size` (64902 = end − start), `effectiveSize` (gap-excluded), `probeCount`, `percentHet` ("0%"), `lrrMedian`, `lrrDev`, `bafDev`, `startCytoBand`, `endCytoBand`, `traditionalNomenclature` ("del(1)(p36.33p36.33)"), `microarrayNomenclature` ("arr[GRCh37] del(1)(p36.33p36.33)(1567924_1632825x1)"), `geneCount`, `genes[]`. The sample-level `microarrayNomenclature` example is verbatim: `arr[GRCh37] del(1)(p36.33p36.33)(1567924_1632825x1),1q31.3(196716621_196801989)x1,3q26.32(175887314_175916348)x1,4p16.3(3562944_3627954)x3,9p24.3q21.11(14326_69444970)x3,14q11.1q11.2(19021424_20422182)x3 hmz,16p11.2q11.2(31382412_46974423)x2 hmz,17q21.31(44225485_44342404)x1,17q25.3(81083660_81152211)x3`. It mixes a nonstandard first element with ISCN-2020 `band(start_end)xN` elements. LOH is written `x2 hmz`, and gain-LOH `x3 hmz`. The traditional form lists only del and dup: `del(1)(p36.33p36.33),…,dup(9)(p24.3q21.11),…`, comma-joined with no LOH. ### 2.14 Methylation files - `_ctrl.tsv.gz`: address, type, color channel, name, probe ID, and raw MG, MR, UG, UR (methylated green and red, unmethylated green and red). - `_cgs.tsv.gz`: per CG, the beta value, M-value and detection p-value. - `sample_qc_summary.{tsv,xlsx}`: `Sentrix_ID` (12-digit), `Sentrix_Position`, `Sample_ID` (optional), user metadata, `restoration`, `staining_green`, `staining_red`, `extension_green`, `extension_red`, `hybridization_high_medium`, `hybridization_medium_low`, `target_removal1`, `target_removal2`, `bisulfite_conversion1_green`, `bisulfite_conversion1_background_green`, `bisulfite_conversion1_red`, `bisulfite_conversion1_background_red`, `bisulfite_conversion2`, `bisulfite_conversion2_background`, `specificity1_green`, `specificity1_red`, `specificity2`, `specificity2_background`, `nonpolymorphic_green`, `nonpolymorphic_red` (the doc counts these as 21 controls), `avg_green_raw`, `avg_red_raw`, `avg_green_norm`, `avg_red_norm` (after dye-bias correction plus noob), `ScanTime` (MM/DD/YY HH:MM), `NProbes`, `NPassDetection`, `prop_probes_passing`, `passQC` (1/0), `failCodes` (list). - `pcs.tsv.gz`: a blank-named first column (barcode_position), `principal component 1`, `principal component 2`, `Sample_Group` (NA if absent). - `probes.csv`: `Probe_ID` (IlmnID, or `ctl_[AddressA_ID]` for controls), `U` (AddressA_ID), `M` (AddressB_ID), `col` (R/G for Infinium I, NA otherwise; the doc misprints this as "For Infinium I probes, this column will be NA", which should read Infinium II). - `controls.csv`: `Address, Type, Color_Channel, Name, Probe_ID`. ### 2.15 QC metrics files - `controls.raw_metrics.csv`: `SampleId, Category (Staining, Extension, Hybridization, Stringency, Non-Polymorphic, …), Beadtype, Control (e.g. "Biotin (High)", "DNP (Bgnd)", "String (PM)"), Color, GreenIntensity, RedIntensity`. There is one row per control-probe measurement per sample (long format). - `controls.qc_metrics.csv`: `SampleId, ImagingDate, SamplePlate, SampleWell, ScannerId, ScannerVersion` (all taken from IDAT metadata), then the metric columns: `RestorationQC, StainingGreenQC, StainingRedQC, ExtensionGreenQC, ExtensionRedQC, HybridizationHighMediumQC, HybridizationMediumLowQC, TargetRemovalIQC, TargetRemovalIIQC, StringencyQC, NonSpecificBindingGreenQC, NonSpecificBindingRedQC, NonPolymorphicGreenQC, NonPolymorphicRedQC, SpecificityIGreenQC, SpecificityIRedQC, SpecificityIIQC, SpecificityIIBgQC, BisulfiteConversionIGreenQC, BisulfiteConversionIGreenBgQC, BisulfiteConversionIRedQC, BisulfiteConversionIRedBgQC, BisulfiteConversionIIQC, BisulfiteConversionIIBgQC`. Columns that do not apply to the array type are omitted. ### 2.16 QC report `_QC_Report_YYYY_MM_DD.html` has these dashboards: Control Dashboard, Automated QC, Sample QC Heatmaps, Trend Analysis, and a QC Metric Config menu for customizing thresholds. It is distributable and works offline. Locally one report can combine multiple datasets for cross-run trends; the cloud version covers one dataset. `_QC_table_YYYY_MM_DD.` has one row per sample: functional metrics when available, derived control metrics, and raw control-probe intensities (wide format). --- ## 3. Semantic rules ### 3.1 Genotype merging (multi-allelic / duplicate probes, MAV) 1. Probes at the same chromosomal coordinate (tri-allelic loci or duplicate sites) collapse into **one VCF record**. 2. The algorithm enumerates every genotype over the union of queried alleles. It removes genotypes inconsistent with any assay's call, "respecting design limitations". Infinium II cannot distinguish A/T or C/G, so an Inf II `[A/G]` AA call is consistent with AA, AT and TT. 3. Exactly one survivor gives a call. Zero survivors (inconsistent) or more than one (ambiguous) gives a NoCall `./.` plus a logged warning. 4. **A NoCall assay does not exclude anything**; it is presumed to be probe failure. It therefore often causes ambiguity. Example: Inf I [T/A]→AA plus Inf II [A/G]→NC gives NoCall, not AA. 5. With `--use-infI-nc-info` (v1.4), an **Infinium I no-call is read as missing signal**. This rules out the alleles that assay queries if other assays do not show them. It raises call rate and false positives, and it needs a cluster file that clusters "zeroed-out" MAV assays. 6. Merged records carry **GT and GS only**, where GS is the minimum GS across the assays. They have no BAF or LRR. 7. REF/ALT come a priori from the manifest and reference. ALT can list several alleles, which allows GTs like `0/2`, `1/3`, `2/3`. 8. The ID lists all probe names, joined by `,` for SNPs and `;` for indels. 9. Indels can be "flipped": a manifest deletion REF=ATCG/ALT=A may be written as an insertion REF=A/ALT=ACTG (as printed). The GT is adjusted to match, so the VCF stays correct. ### 3.2 Ploidy - The SNV VCF is haploid on Y, MT and non-PAR X for males. For females, calls on Y are presumably haploid too. - Cyto and PGx CN: `##OverallPloidy` is the arithmetic mean ploidy across the genome, accounting for event lengths. The baseline differs by sex (≈1.95 for a male per the examples). `copyNumberMedian` is the length-weighted median CN. ### 3.3 BAF/LRR orientation The VCF writes BAF and LRR with **REF = A and ALT = B relative to the reference genome**. It may therefore report 1 − BAF compared with the legacy tool, which orients by manifest A/B. The page does not say that LRR changes (LRR does not depend on allele order). GS does not depend on orientation. ### 3.4 Filters - SNV VCF: FILTER is always PASS; use GS for quality. - PGx CNV VCF: `Q7` (QUAL<7) and `SampleQuality` (sample LogRDev > 0.2). - Cyto VCF: `Q10` (QUAL<10), `SampleQuality` (LogRDev > 0.2), `LowFraction` (mosaic MF below a threshold whose value is not given). - Cyto reporting gate `minQual` = 20, plus per-type minimum probe counts and sizes (see §5). ### 3.5 QUAL - Cyto: `QUAL = −10·log10(1 − P(called state))`, capped at 60, minimum 0. - PGx CNV: "Phred-scaled … capped at 60 and the minimal value is 0", written as an integer. - Cyto JSON `qualityScore` is a float (60.0). ### 3.6 CN encodings - PGx CN is an integer from 0 to 5, where "CN=5 indicates 5 or 5+". It is labelled "for imprecise events". - Cyto CN is an integer from 0 to 4, where "CN=4 indicates 4 or 4+". - ALT is symbolic: `` in PGx; ``, `` and `` in cyto. DUP is described as a "Copy number gain heterozygous region". - NR is the aggregated normalized intensity, ≈ CN/2 relative to diploid. - Microarray ISCN uses `xN` for the copy number and appends `hmz` for LOH. ### 3.7 LOH types There are three `LOHTYPE` values. `AOH` is germline LOH, copy-neutral or gain, and CN can be 2 or 3. `CNLOH` is somatic copy-neutral LOH. `GAINLOH` is somatic gain LOH. There are separate minimums for GAINLOH (300 probes, 500 kb) and for mosaic LOH ("Mloh": 500 probes, 3000 kb). The per-chromosome LOH counts exclude GAINLOH; the sample-level counts include it. ### 3.8 Mosaicism - The `MOSAIC` flag and `MF` (estimated mosaic fraction, a float) mark a mosaic call; MF is `.` when not mosaic. - The ID prefix is `Mosaic`. - `LowFraction` is the FILTER for an MF below a threshold. - `HIGHFRACTION` is the "Indicator for promoted events with high mosaic fraction". A mosaic event with a high MF is promoted to a regular (constitutional) event but keeps the flag. - There are separate minimum sizes and probe counts for mosaic DEL, DUP and LOH. - Each chromosome has `hasMosaicism`, `numberMosaic` and `medianMosaicFraction`. ### 3.9 ISCN notation - `iscnVersion` is "ISCN 2020". - Traditional form: `del(chr)(bandStartbandEnd)` / `dup(...)`, comma-joined across events. - Microarray form: an `arr[GRCh37]` or `arr[GRCh38]` prefix, then per event `(_)x`, with ` hmz` for LOH. Coordinates are 1-based. - Cytobands for the start and end come from the annotation DB. ### 3.10 Gene annotation Genes are listed when a **transcript's coordinates intersect** the event. Names are HGNC symbols from RefSeq, plus Ensembl gene IDs that have no matching symbol. Clone-style names like `RP11-345P4.10` also appear in the hg19 example. `geneCount` is the length of the list. ### 3.11 Effective size A gap is a probe spacing more than 150× the median probe spacing. Each gap is replaced by the median spacing, and `effectiveSize` is computed with those replacements. ### 3.12 Star-allele scoring and ranking - `Score`/`qualityScore` is a probability from 0 to 1. It depends on SNV and CNV evidence, input quality (GS) and population allele frequencies. `Raw Score`/`rawScore` is the same without population frequency. - Solutions are ranked by qualityScore; rank 1 is the top. - **Alternative solutions with qualityScore ≥ 50% of the top score are "feasible"** and all of them are listed in `genotype`, `;`-separated. - Alternatives also deserve consideration when their supporting variants have GS < 0.15; the guidance is to check those clusters for shift. - Array-equivalent alleles are collapsed, and the one with the highest population frequency is reported. - There are background alleles for SVs. - The copy-number solution is given per sub-region: CYP2D6 p5, intron.2 and exon.9; CYP2A6 exon.1 and intron.7; and whole-gene regions for GSTM1, GSTT1, UGT2B17, SULT1A1 and CYP2E1. - Allele nomenclature includes tandem and hybrid forms: `*36x2+*83+*10`, `*13+*68x2+*4`, `*1x3N` (≥3 copies, N = unknown count), `*4.013` (suballele). ### 3.13 Phenotype - `PGx Guideline` / `pgxGuideline` is CPIC or DPWG, chosen by the user. - The activity score is numeric or "n/a". - The phenotype annotation is a string such as "Normal Metabolizer", "Decreased Function", "CYP2D6 Indeterminate", "Ivacaftor non‑responsive", or "n/a". - Database sources are listed with their versions. ### 3.14 Methylation outputs - Beta runs from 0 (unmethylated) to 1 (methylated). The M-value is a log transform of beta (log2(β/(1−β)), which the page does not spell out). - Detection p-values above 0.05 are recommended for exclusion. - Processing is Sesame-based: dye-bias correction plus noob normalization. - PCA uses the betas of all analytical probes. **When there are more than 100 samples, a random subset of 10,000 probes is used.** Plots are colored by Sample_Group. - A sample passes when all 21 control metrics pass. ### 3.15 Control-probe QC formulas (genotyping) `bkg_grn` = max green of Extension(A) and Extension(T). `bkg_red` = max red of Extension(C) and Extension(G). x = 3000. | Metric | Genotyping | Methylation | |---|---|---| | RestorationQC | Restoration green/(bkg_grn+x) | same | | StainingGreenQC | Biotin(High) g / Biotin(Bgnd) g | same (Bkg) | | StainingRedQC | DNP(High) r / DNP(Bgnd) r | same | | ExtensionGreenQC | min Ext(C,G) g / max Ext(A,T) g | same | | ExtensionRedQC | min Ext(A,T) r / max Ext(C,G) r | same | | HybridizationHighMediumQC | Hyb(High) g / Hyb(Medium) g | same | | HybridizationMediumLowQC | Hyb(Medium) g / Hyb(Low) g | same | | TargetRemovalIQC | (bkg_grn+x)/Target Removal g | (bkg_grn+x)/TR1 g | | TargetRemovalIIQC | — | (bkg_grn+x)/TR2 g | | StringencyQC | String(PM) r / String(MM) r | — | | NonSpecificBindingGreenQC | (bkg_grn+x)/max NSB(Bgnd) g | — | | NonSpecificBindingRedQC | (bkg_red+x)/max NSB(Bgnd) r | — | | NonPolymorphicGreenQC | min NP(C,G) g / max NP(A,T) g | same | | NonPolymorphicRedQC | min NP(A,T) r / max NP(C,G) r | same | | SpecificityI Green/Red, II, IIBg; BisulfiteConversion I Green/GreenBg/Red/RedBg, II, IIBg | — | see §2.15 / L1233–1242 (GT Mismatch 1–3 green, 4–6 red; BS Conv I C1–C2/U1–U2 green, C3–C5/U3–U5 red) | The older methylation-summary table (L1080) gives slightly different wording: the red bisulfite-conversion background is "Background/(Highest value of U4, U5, or U6)", and Specificity II background uses S1–S4. These differ from the newer table's U3–U5. ### 3.16 Sample summary metrics Call rate = #Calls/(#No_Calls+#Calls). The autosomal call rate and LogRDev (LRR SD) are also reported. Sex is F or M; "Male" in cyto; "U" is presumably possible. TGA_Ctrl_5716 Norm R < 1 means the targeted gene amplification failed. Cyto adds `bafDev` per genotype class, `medianLogRDev`, `percentLOH`, the LOH counts, `overallPloidy` and `copyNumberMedian`. --- ## 4. Messages ### 4.1 Genotyping, CN and PGx (terminal) General scenarios: - The manifest used to make the GTC differs from the manifest of the CN model. - The FASTA and the FASTA index do not match. - Indel processing failed. - The input folder lacks required files. - An input file is corrupt. | Message (verbatim) | Type | Trigger | |---|---|---| | `Failed to normalize and gencall sample: {sample_id}, it will be skipped. Error: The given key '{loci_id}' was not present in the dictionary.` | Warning | The manifest and cluster file do not match (the EGT was built from a different manifest) | | `Reference allele is not queried for locus: {identifier}` | Warning | The true reference allele is not among the manifest alleles (common for MNVs) | | `Skipping non-mapped locus: {identifier}` | Warning | Chr = 0 or no position | | `Skipping intensity only locus: {identifier}` | Warning | Intensity-only probe | | `Skipping indel: {identifier}` | Warning | Indel context (deletion/insertion) could not be determined | | `Failed to process entry for record: {identifier}` | Warning | The indel's reference allele could not be determined | | `Incomplete match of source sequence to genome for indel: {identifier}` | Warning | The indel does not map properly to the reference | | `Failed to combine genotypes due to inconsistencies - exm1068284 (InfiniumII): TT, ilmnseq_rs1131690890_mnv (InfiniumII): AA, rs1131690890_mnv (InfiniumII): AA` | Warning | MAV merge conflict gives `./.`. Format: ` (): ` joined by `, ` | | `Failed to combine genotypes all were No Calls - exm1068284 (InfiniumII): NC, …` | Warning | Every merged probe is NC | | `Cluster file ({GTC.egt}) is not the same as CN Model Cluster file ({CN_Model.egt}).` | Warning | The EGT used for the GTCs differs from the one used when the CN model was trained | | `{numPassingSamples} sample(s) passed QC.` / `Requires at least {minPassingSamples} samples to proceed.` | Error | Too few high-quality samples for batch-dependent CNV calling (the minimum is not given here) | | `Invalid manifest file path {manifestPath}` | Error | The manifest was not found | | `Failed to load cluster file: {e.Message}` | Error | The EGT is corrupt or an unsupported version | | `System.IO.EndOfStreamException: Unable to read beyond the end of the stream.` | Error | GTC read failure (a known i18n issue). This reveals that the tool is a .NET implementation | ### 4.2 Methylation logs | Log | Message | Type | Trigger | |---|---|---|---| | write_samplesheet.log | `No IDATs found` | Error | No IDATs provided | | format_samplesheet.log | `No samples in sample sheet` | Error | The sample sheet is empty | | format_samplesheet.log | `Sample sheet not correctly formatted` | Error | Not CSV, or header lines do not start with "<" (sic) | | format_samplesheet.log | `beadChipName and sampleSectionName columns are required for the sample sheet.` | Error | Required columns missing | | format_samplesheet.log | `Warning: samples have duplicate Sample_ID` | Warning | Duplicate `_` values; the duplicates are dropped | | convert_manifest_ilmn_sesame.log | `Missing control probes in manifest` | Error | No `[Controls]` line | | convert_manifest_ilmn_sesame.log | `Probe section not found` | Error | No `[Assay]` line | | convert_manifest_ilmn_sesame.log | `Missing required columns: IlmnID, AddressA_ID, AddressB_ID, Color_Channel` | Error | A column is missing from the Assay section | | convert_manifest_ilmn_sesame.log | `Controls not formatted correctly. Must have 4 columns (Address,Type,Color_Channel,Name)` | Error | Bad Controls section | | run_sesame_gs.log | `Missing sample: ` | Error | IDATs missing for a sample | | run_sesame_gs.log | `No scan time available` | Warning | The IDAT has no scan time | | run_sesame_gs.log | `Prep failed` | Error | Dye-bias correction or noob failed | | run_sesame_gs.log | `Warning: missing control probe types` + list | Warning | The metric is set to NA | | run_sesame_gs.log | `Warning: missing control probe names` + list | Warning | The metric is set to NA | | qc.log | `No features, skipping PCA plot` | Warning | No beta is common to all samples (e.g. a sample with no signal) | --- ## 5. Numeric thresholds | Value | Context | |---|---| | 60 / 0 | QUAL cap and floor (PGx CNV and cyto) | | QUAL < 7 → `Q7` | PGx CNV filter | | QUAL < 10 → `Q10` | Cyto filter | | minQual 20 | Cyto reporting | | LogRDev > 0.2 → `SampleQuality` | Both CNV VCFs | | CN=5 means 5+ (PGx); CN=4 means 4+ (cyto) | CN caps | | DEL ≥10 probes and ≥25 kb; DUP ≥10 and ≥50 kb; LOH ≥500 and ≥3000 kb; GAINLOH ≥300 and ≥500 kb; mosaic DEL 10/25 kb, mosaic DUP 10/50 kb, mosaic LOH 500/3000 kb | Cyto reporting minimums (example defaults) | | 150× median probe spacing | Gap definition for effectiveSize | | 1 Mbp, 8 Mbp | LOH count bins | | TGA_Ctrl_5716 Norm R < 1 | TGA failure | | ≥ 50% of top qualityScore | Alternative star-allele solutions are listed | | GS < 0.15 | Low-confidence supporting variant; consider alternatives | | Score range 0–1 | PGx scores | | > 1 M loci and > 100 samples → 50+ GB | Final report size warning | | detection p > 0.05 | Methylation exclusion recommendation | | > 100 samples → 10,000 random probes | Methylation PCA | | x = 3000 | Background offset | | Methylation defaults (EPIC) | restoration 0 (1 with FFPE restore); staining 5; extension 5; hybridization 1; target removal 1; BS conv I/II 1 (**MSA 0.5** for the Bg metrics and BS-II); specificity 1; nonpolymorphic green 5 (**MSA 2.5**), red 5 (**MSA 3**) | | 5 pM / 1 pM / 0.2 pM | Hyb control target concentrations | | 21 | Methylation control metrics | | 12 digits | Sentrix barcode | | 8 decimals (summary), 4 (GS), 8 sig. digits (BAF/LRR) | Output precision | --- ## 6. Hints about the underlying algorithms - **CNV QUAL is a posterior** probability of the called state, so the model is probabilistic (HMM or mixture). The Phred cap of 60 means P ≥ 1−10⁻⁶ is saturated. - **CNV calling is batch dependent.** It needs a minimum number of QC-passing samples and a "CN model" trained from GTCs with a given cluster file, which is tied to a specific manifest and cluster file (both are checked). The model is "robust to minor cluster file updates". PGx CN is therefore a trained, batch-normalized per-region caller: NR is normalized intensity against a reference, much like a batch median. - `##GCCorrect=True` and the JSON `gcCorrect` show that GC-wave correction is an option. - `OverallPloidy` is a length-weighted mean CN over the genome. `copyNumberMedian` is the length-weighted median. - Cyto LOH uses heterozygosity (`percentHet`) and `bafDev`. Mosaic MF estimation is from BAF/LRR shift. There is a promotion rule that turns mosaic events into regular calls at high MF. LowFraction events are kept but filtered. - Separate minimums for germline, gain-LOH and mosaic events suggest distinct detection passes (constitutional and mosaic) that are then merged. - **Star-allele caller**: probabilistic scoring from SNV genotypes (GS as quality), CNV region CNs, and population frequency priors. Variant copy number per supporting variant means a CN-aware genotype model, with BAF used to find how many copies carry the ALT. Array-equivalent collapsing is done on allele definitions restricted to array content. There is an intermediate `.dat`. - **MAV merging** uses a consistency-filter algorithm (see §3.1). The failure-message format exposes per-probe assay type. - The software stack appears to be .NET for genotyping (`System.IO.EndOfStreamException`, "The given key … was not present in the dictionary" is a .NET KeyNotFoundException message) and R/Sesame for methylation (`run_sesame_gs`, `convert_manifest_ilmn_sesame`). The cyto booleans (`True`) are Python-style. - The GTC includes sample QC metrics, which is where the summary metrics come from. - The BAF interpolation matches the classic legacy definition, and the LRR for intensity-only loci uses the cluster mean. --- ## 7. Gap analysis vs the SimurgArray plan (§8, §9, §10, §11, §13) Legend: **MISSING** means the plan has no equivalent. **DIFF** means the plan specifies it differently. **UNDERSPEC** means the plan mentions it without the detail needed. ### 7.1 SNV VCF (§8.3) 1. **DIFF: FILTER and QUAL.** The plan does not say what FILTER holds. Proposed spec: *"The per-sample genotype VCF sets QUAL to `.` and FILTER to `PASS` for all records by default; call quality is carried only in FORMAT/GS. An optional `--filter-gs ` may add a `LowGS` FILTER (off by default, for compatibility)."* 2. **DIFF: INFO fields.** The plan adds `ASSAY_TYPE, NORM_BIN, GENTRAIN, CLUSTER_SEP, ILMN_STRAND, PROBE_IDS`; the reference output has no INFO. Proposed spec: *"INFO annotations are opt-in (`--info-fields`); the default output has INFO `.` so the file compares byte-for-byte with other array-to-VCF converters."* 3. **UNDERSPEC: MAV handling.** Proposed spec: *"Merged records emit FORMAT `GT:GS` only (BAF and LRR omitted from the FORMAT string for that record, not written as `.`); GS = min over contributing assays. A no-call assay constrains nothing by default. Flag `--infI-nocall-informative`: an Infinium-I no-call excludes that assay's queried alleles when no other assay supports them (documented to raise both call rate and false positives, and to require a cluster file that clusters the zeroed-out assays). Each merge failure writes a WARNING `Failed to combine genotypes due to inconsistencies - (): , …` or `… all were No Calls - …`, which also goes to the audit table."* 4. **MISSING: indel REF/ALT flip note.** Proposed spec: *"When the manifest describes a deletion that normalizes to an insertion against the reference (or vice versa), write the reference-consistent representation and remap GT; record `flipped_indel=true` in the audit."* 5. **MISSING: skip taxonomy and warnings.** Proposed spec: *"Loci are skipped with one warning each (rate-limited, all listed in the audit): non-mapped (chr 0/pos 0), intensity-only, indel context undeterminable, indel reference allele undeterminable, incomplete source-sequence match, reference allele not among queried alleles (the record is still written with the queried alleles as ALT)."* 6. **MISSING: header meta keys.** Proposed spec: *"The header includes `##genomeBuild=<37|38>`, and only contigs present in the reference (including alt contigs when used); contig order follows the reference .fai."* 7. **UNDERSPEC: precision.** Proposed spec: *"GS written with 4 decimals, BAF/LRR with 8 significant digits."* 8. **MISSING: a PLINK conversion recipe and multi-allelic split guidance.** Proposed spec: *"The documentation shows how to convert via a bcftools merge, then `bcftools norm -m-`, then plink; `simurg export plink` splits multi-allelic sites by default (`--keep-multiallelic` to drop them instead)."* ### 7.2 Per-sample summary (§13 `sample_summary.tsv`) 9. **DIFF/MISSING columns.** Proposed spec: *"The sample summary always contains `sample_id` (barcode_position), `sample_name` (resolved by the sample-name rule), `sample_folder`, `autosomal_call_rate`, `call_rate`, `logr_dev`, `sex_est` (F/M/U), then every sample-sheet column passed through verbatim in sheet order. If the manifest contains the targeted-amplification control probe (configurable name list), add `_norm_r` and flag `tga_fail` when it is < 1.0. Failed samples stay as rows with a `status=FAILED` and `fail_reason`. JSON form: an array of flat objects with the same keys."* ### 7.3 Final report and locus summary 10. **DIFF: Final Report.** The plan treats the legacy final report as read-only (§6.7), though `simurg export finalreport` exists. Proposed spec: *"`export finalreport` writes a long-format table with columns SNP Name, SNP, Sample ID, Allele1/2 Top, Allele1/2 Forward, Allele1/2 Plus, GC Score, GT Score, Log R Ratio, B Allele Freq, Chr, Position. It warns when loci × samples > 1e8 (a very large file)."* 11. **UNDERSPEC: locus summary columns.** The plan (§7.9) has the formulas. Add: *"Column set and order: Locus_Name, Locus_ID, n_nocalls, n_calls, call_freq, aa_freq, ab_freq, bb_freq, minor_freq, gentrain_score, gc50, gc10, het_excess, chi_test_p100, cluster_sep, AA/AB/BB θ mean and sd, AA/AB/BB R mean and sd, plus_minus_strand (+/−/U)."* Note that the page defines het excess as `(f_ab − 2pq)/(2pq+ε)`; the plan's k-regularized form must be documented as equivalent or deliberately different. ### 7.4 QC module (§9) 12. **DIFF: file layout.** The plan names `controls_raw.tsv` and `controls_qc.tsv`. Proposed spec: *"`controls.raw_metrics.csv` is long format (SampleId, Category, Beadtype, Control, Color, GreenIntensity, RedIntensity). `controls.qc_metrics.csv` has one row per sample with IDAT-derived metadata (ImagingDate, SamplePlate, SampleWell, ScannerId, ScannerVersion) followed by metric columns; columns that do not apply to the array type are omitted. Both files are produced automatically by genotyping and by a standalone `qc controls` command."* The plan's §6.1 must make sure the IDAT parser extracts these metadata fields. 13. **DIFF: NSB red formula.** The plan writes "(bkg + x)/max NSB(Bgnd)". Spec: *"NSB green = (bkg_grn + x)/max NSB(Bgnd) green; NSB red = (bkg_red + x)/max NSB(Bgnd) red; bkg_grn = max green of Extension A/T; bkg_red = max red of Extension C/G."* Add a Restoration metric and a separate TargetRemovalII (methylation only). 14. **DIFF: thresholds.** The plan gives hyb 1.25 and NP 5. Methylation defaults are hyb 1, target removal 1, staining 5, extension 5, NP 5 (lower on some chips: 2.5 green / 3 red), BS-conv Bg 0.5 on some chips, restoration 0 (1 with FFPE restore). Proposed spec: *"Thresholds are per-array-family YAML profiles; the report records which profile was used; pass = all enabled metrics pass; `fail_codes` lists the failing metric names."* 15. **MISSING: QC table and report naming.** Proposed spec: *"`qc report` writes `_QC_Report_YYYY_MM_DD.html` (offline, dashboards: controls, automated pass/fail, per-sample metric heatmap, trend over runs, in-page threshold editor) and `_QC_table_YYYY_MM_DD.{xlsx|csv}` (`--output-format`, default xlsx) holding functional metrics, derived control metrics and raw control intensities in wide form. Several datasets may be combined into one report for trend analysis."* ### 7.5 Copy-number and cyto (§10, §13 `cnv.vcf.gz`) 16. **DIFF: cyto CNV VCF schema.** The plan's "`//` with CN, MF, QUAL" is far too thin. Proposed spec: *"Per-sample `.cnv.vcf.gz` (+tbi), VCF 4.1. Header: `##genomeBuild`, `##product=`, `##reference`, `##OverallPloidy=`, `##GCCorrect=`. ALT: DEL (copy loss), DUP (copy gain), LOH (absence/loss of heterozygosity). FORMAT: GT (fixed `1/1` placeholder for events, documented), CN (integer, top value means that value or more; cap configurable, default 4), NR (aggregated normalized intensity ≈ CN/2), LRD (SD of LRR within the event), MF (mosaic fraction; `.` if not mosaic). INFO: SVLEN (positive length = END − POS), PROBE (probe count), END, MOSAIC (flag), LOHTYPE (AOH = germline copy-neutral or gain; CNLOH = somatic copy-neutral; GAINLOH = somatic copy-gain), HIGHFRACTION (flag: mosaic event promoted to constitutional because MF is high). FILTER: `Q10` (QUAL < 10), `SampleQuality` (sample LRR SD > 0.2), `LowFraction` (MF below a configurable threshold). POS = event start − 1 (padding base), REF = `N`. ID = `:::` with a `Mosaic` prefix for mosaic events. QUAL = min(60, max(0, round(−10·log10(1 − P(state)))))."* Also document the SVLEN sign choice: positive, per the reference behaviour, with an option for spec-correct negative values for DEL. 17. **MISSING: reporting gates.** Proposed spec: *"Events are reported only if QUAL ≥ minQual (default 20) and they meet the per-type minimums: DEL 10 probes/25 kb, DUP 10/50 kb, LOH 500/3000 kb, GAINLOH 300/500 kb, mosaic DEL 10/25 kb, mosaic DUP 10/50 kb, mosaic LOH 500/3000 kb. All are configurable and echoed in the annotation JSON."* The plan has no GAINLOH or mosaic minimums and no minQual 20. 18. **MISSING: cyto annotation JSON.** The plan §10.4 has no output schema. Proposed spec: *"`.cyto.json` contains the sample-level fields annotation DB name, software version, reference, annotation type (constitutional|oncology), build, database sources and versions, ISCN version, sample id and name, gcCorrect, every threshold, overallPloidy, callRate, logRDev, medianLogRDev, bafDev{AA,AB,BB}, numLOHOver1M, numLOHOver8M, totalSizeLOHOver1M (these three include gain-LOH), copyNumberMedian (length-weighted), percentLOH, sexEstimate, traditional and microarray ISCN strings. Per-chromosome block: id, size, percentHet, hasMosaicism, lrrMedian, lrrMean, lrrDev, the LOH counts (excluding gain-LOH), percentLOH, copyNumberMedian, copyNumberMean, min and max LRR, medianMosaicFraction, counts of del, dup, LOH, gain-LOH and mosaic events. Per-event block: id, chrom, start (0-based), end, callType, mosaicState, mosaicFraction, copyNumber, qualityScore, size, effectiveSize (gaps > 150× median spacing replaced by the median spacing), probeCount, percentHet, lrrMedian, lrrDev, bafDev, start and end cytobands, traditional and microarray ISCN, geneCount, genes (transcripts intersecting the event: RefSeq symbols plus unmatched Ensembl IDs)."* Use real JSON types (numbers not strings, null not ".") and document this as a deliberate deviation. Offer a `--compat-json` flag for string sentinels. 19. **UNDERSPEC: ISCN.** Proposed spec: *"Microarray ISCN (2020): `arr[GRCh3x] (_)x[ hmz]`, comma-joined; LOH gives `x2 hmz` (or `x3 hmz` for gain-LOH); mosaic events use the ISCN mosaic notation `x1~2` or similar (must be defined). Traditional: `del()()`/`dup(...)` for CNVs only."* 20. **MISSING: batch minimum and model consistency checks.** Proposed spec: *"CN calling aborts with an error ` sample(s) passed QC. Requires at least samples to proceed.` when too few samples pass QC (m configurable, default = the batch-reference minimum). A CN model records the manifest and cluster-file hashes; a mismatch with the input GTCs produces a warning (cluster file) or an error (manifest)."* 21. **MISSING: CN summary file.** Proposed spec: *"`cn_sample_summary.tsv`: sample_id, sample_name, sample_folder, logr_dev, overall_ploidy, gc_correct, n_events per type, pass/fail."* 22. **DIFF: sex and ploidy baseline.** Spec: *"overallPloidy = length-weighted mean CN over all reported contigs, including sex chromosomes, so males are ≈1.95 without events."* ### 7.6 PGx (§11, §13 `pgx.json`) 23. **MISSING: a dedicated targeted-CN VCF.** Proposed spec: *"`.pgx_cn.vcf.gz` (+tbi) has exactly one record per predefined target region (always emitted): ID `CNV::::`, REF `N`, ALT ``, INFO CNVLEN (END−POS+1), PROBE, END, SVTYPE=CNV; FORMAT `CN:NR` with no GT; CN integer 0–5 (5 = 5+); QUAL Phred capped at 60; FILTER `Q7` (QUAL < 7) and `SampleQuality` (LRR SD > 0.2); header `##OverallPloidy`, `##GCCorrect`, `##genomeBuild`. The region catalogue (CYP2D6 5′ region/intron 2/exon 9; CYP2A6 exon 1/intron 7; whole-gene GSTM1, GSTT1 (alt contig on GRCh38), UGT2B17, SULT1A1, CYP2E1) is a versioned BED."* The plan mentions only CYP2D6 sub-regions. 24. **MISSING: `cn_batch_summary.csv`.** Proposed spec: *"Per target region: counts of gain (CN > 2), loss (CN < 2) and neutral (CN = 2) samples, plus an `All Target Regions` totals row."* 25. **MISSING: PGx tabular outputs.** Proposed spec: *"The batch-level PGx directory contains `diplotype_summary.csv`, `diplotype_annotations.csv`, `supporting_variants.csv`, `unresolved_variants.csv` and `variants_not_covered_on_array.csv`, with columns as in §2.6 of this analysis (renamed to snake_case). Each starts with `#`-prefixed metadata lines (build, allele DB version, phenotype DB version, mapping-file hash)."* 26. **UNDERSPEC: star-allele JSON.** Proposed spec: *"Per-sample `.pgx.json`: tool version, build, allele and phenotype DB sources with versions, mapping-DB file name and hash, guideline (CPIC|DPWG), sample id, and per gene: callType (star allele|variant), genotype (all feasible top solutions `;`-joined), array-equivalent alleles per allele, activity score, phenotype, qualityScore (with population prior) and rawScore (without), supporting variants (chrom, pos, ref, alts, gt, gs, baf, variant CN, hgvs, rsid, probe ids, allele ids), no-call variants, alleles impacted by no-calls, ranked candidateSolutions (rank, genotype, scores, alleles with their supporting and unresolved sites, array-equivalents, background allele, variantChange, copyNumberRegions and copyNumberSolution), variantsNotCoveredOnArray, allelesTested. Use consistent JSON types (numbers as numbers, no stray key punctuation)."* 27. **MISSING: ranking rules.** Proposed spec: *"Solutions within ≥ 50% of the top qualityScore are reported as co-top (`;`-joined). Supporting variants with GS < 0.15 are flagged `low_gs` and the gene call gets an `review_alternatives` flag. Among array-indistinguishable alleles, report the one with the highest population frequency and list the rest."* 28. **MISSING: an intermediate calls file** between `star-allele call` and `annotate`. Spec: *"PGx calling is split into `pgx call` (writes `.pgx_calls.parquet`) and `pgx annotate` (phenotype and guideline lookup), so a guideline can be re-annotated without re-calling."* 29. **MISSING: a variant-type PGx call** for SNP-only genes/variants (e.g. rsID-level calls, MT-RNR1). Spec: *"Genes without star-allele definitions are reported as `Type=Variant` with the explicit genotype."* 30. **MISSING: CN-aware variant copy number and BAF use** in star-allele scoring (per-variant CN, BAF ≈ k/CN). The plan §11 checks only "CN-consistent BAF (1/3, 2/3)". Spec: *"For CN = n, compute the expected BAF for k alt copies as k/n and use the likelihood of the observed BAF to assign variant copy number k."* ### 7.7 Methylation (plan v2 item 14: "methylation-array control QC (optional)") 31. **MISSING almost entirely.** If kept in scope, proposed spec: *"Per sample: `__ctrl.tsv.gz` (address, type, channel, name, probe id, raw MG/MR/UG/UR) and `__cgs.tsv.gz` (beta, M = log2(β/(1−β)), detection p). Per batch: `sample_qc_summary.tsv` and `.xlsx` (failing cells highlighted), holding the 21 control metrics, avg raw and normalized green/red, scan time, n_probes, n_pass_detection, prop_probes_passing, passQC (1/0), failCodes. Also `sample_qc_summary.pdf` (a histogram of the passing proportion colored by pass status; PCA on betas, using a random 10,000 probes when n > 100, colored by Sample_Group), `pcs.tsv.gz` (sample, PC1, PC2, Sample_Group or NA), a converted manifest `probes.csv` (Probe_ID, U, M, col) and `controls.csv`. Normalization: dye-bias correction plus noob (an open reimplementation). Detection p > 0.05 is recommended for exclusion."* Also add the log and validation messages: no IDATs, empty sheet, missing columns, duplicate sample (dropped with a warning), missing `[Controls]`/`[Assay]`, missing manifest columns, missing sample, no scan time (warning), prep failure, missing control types (metric = NA), no common features (PCA skipped). ### 7.8 Other 32. **UNDERSPEC: bedGraph.** Specify the columns (`chrom, start0, end, value`), sort order, excluded loci (intensity-only for BAF) and a track line. 33. **MISSING: an error-message catalogue.** Add a table of stable message IDs covering, at minimum: manifest/cluster mismatch (the sample is skipped with a warning), invalid manifest path, cluster-file load failure, truncated GTC, FASTA/.fai mismatch, CN-model manifest or cluster mismatch. 34. **MISSING: legacy GTC compatibility.** Spec: *"GTCs from other tools are accepted for VCF export; CN, cyto and PGx modules require GTCs that carry normalization transforms and the needed intensities, and refuse others with a clear error."* 35. **Test oracles.** Add the reference-output inconsistencies to §20 (known unknowns): LOH example with mismatched CHROM; GAINLOH example with LOHTYPE=AOH; SULT1A1 ID versus POS/END mismatch; string versus number scores; "CN:" key typos; annotationType int versus string. Tests must not encode these as requirements.