Add sanitized Phase-0 research archive under docs/research
Research notes R1-R4, documentation-study files A-E (source of the functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use neutral naming and current project decisions (plan §2).
This commit is contained in:
kemal committed
1 parent
180b6f160a
commit
91e97424c7
12 files changed
+7748
No files matched your search
@@ -0,0 +1,736 @@
|
||||
# E. Release history of the reference genotyping-array product (v1.0 → v1.4)
|
||||
|
||||
> **Archived Phase-0 study (sanitized copy, see `README.md`).** Written 2026-09-27, before the decisions in
|
||||
> `SIMURGARRAY_MASTER_PLAN.md` §2. The analysed vendor product is called "the reference product" and its CLI
|
||||
> `refcli`; its documentation URLs are replaced by `[ref-docs:…]` tokens. Where this file speaks of a clean room,
|
||||
> legal clearance, or running/benchmarking the vendor binary, plan §2 supersedes it: personal research; MIT/BSD/Apache
|
||||
> code may be reused with attribution; GPL tools are subprocess oracles only; vendor binaries are never run for
|
||||
> comparison, inspected or decompiled; oracles are published vendor-produced GTCs, gtc2vcf and BeadArrayFiles.
|
||||
|
||||
Source set (all read in full): `support-and-updates/release-notes.md` + every page under
|
||||
`support-and-updates/release-notes/` (v1.0.0, cloud genotype 1.0.0, cloud methylation-QC 1.0.0/1.0.1, v1.1.0,
|
||||
v1.2.0 + EMGv38 case-creation, v1.3.0 + EMG V100.39, v1.4.0 + EMG V100.40), `support-and-updates/document-revision-history.md`,
|
||||
the archived v1.0 … v1.3 doc trees (release notes, revision history, FAQ, features, local-analysis,
|
||||
input-files, output-files, cloud pages, PGx reference pages), compared by normalised `diff` (links/whitespace/GitBook
|
||||
footer stripped) against the current `product-guides/*.md`, `overview/our-features.md`, `support-and-updates/faq.md`.
|
||||
|
||||
Terminology: "the product" = the vendor tool; executable name `refcli`. "EMG" = the vendor's tertiary interpretation
|
||||
platform. Quotes are verbatim or near-verbatim; `[sic]` marks upstream typos kept on purpose.
|
||||
|
||||
---
|
||||
|
||||
## 0. Timeline at a glance
|
||||
|
||||
| Doc rev | Date | Event |
|
||||
|---|---|---|
|
||||
| 01 | Dec 2023 | v1.0.0 local release (genotype, PGx CNV, star-allele) |
|
||||
| 02 | Mar 2024 | Cloud Genotyping pipeline 1.0.0 |
|
||||
| 03 | May 2024 | Cloud Methylation-QC pipeline 1.0.0; doc fix: CNV VCF example CN=4 → CN=5 |
|
||||
| 04 | Sep 2024 | v1.1.0 (EX PGx chips, JSON standardisation, cluster/CN-model decoupling) |
|
||||
| 05/06 | Feb 2025 | v1.2.0 (cytogenetics CNV/LOH, BAF bedGraph, `pgx` root command); notes updated |
|
||||
| 07 | Jun 2025 | v1.2.0 notes updated: gtc-to-bedgraph LRR smoothing bug; "Bedgraph Smoothing window size" disabled in cloud UI; Cyto + EMG pipeline 1.2.0 (auto case creation, EMG v38) |
|
||||
| 08 | Aug 2025 | v1.3.0; "CNV and LOH Calling" renamed "Cytogenetics analysis" |
|
||||
| 09 | Oct 2025 | Cyto + EMG cloud pipeline 1.3.0 (EMG V100.39.0) |
|
||||
| 10 | Feb 2026 | Methylation-QC cloud 1.0.1 hot-fix |
|
||||
| 11 | Mar 2026 | Vendor cloud platform "Planned Analyses" (hub v7.44.0); cloud section restructured |
|
||||
| 12 | May 2026 | v1.4.0 (QC report, GAINLOH, PGx redesign, MAV Inf-I no-call option, .NET 9) |
|
||||
| 13 | Jun 2026 | Cyto + EMG cloud pipeline 1.4.0 (EMG V100.40.0) |
|
||||
| 14 | Aug 2026 | vendor cloud data layer rebranded; vendor cloud UI update; PGx coverage page & v1.4 limitations updates |
|
||||
|
||||
Deprecation (added Q1 2026): cloud pipeline versions *Methylation QC 1-0-0, PGx Star Allele Annotation 1-0-0,
|
||||
Genotyping 1-0-0, PGx CNV Calling 1-0-0* are deprecated.
|
||||
|
||||
---
|
||||
|
||||
## 1. Per-version detail
|
||||
|
||||
### 1.1 v1.0.0 (local) — December 2023
|
||||
|
||||
**Highlights**
|
||||
- Improved star-allele calling accuracy for GDA-ePGx chips.
|
||||
- Star-allele calls reported with quality scores.
|
||||
- "Missing variant reporting to improve data quality."
|
||||
|
||||
**New features**
|
||||
- Star-allele calling for genes in the coverage list. "For in-silico datasets, call rate ≥99%, diplotyping accuracy ≥ 90%";
|
||||
reports hybrid star alleles and allele-specific copy number.
|
||||
- Quality score estimating confidence; star-allele call rate increased through "more robust error tolerance and
|
||||
missing data tolerance"; supporting and missing variants listed; quality score accounts for missing data.
|
||||
- Ranked alternative star-allele solutions with "quality score (negative log likelihood)".
|
||||
- Function annotation: metabolizer + function for CPIC and DPWG; activity scores for CYP2C9, CYP2D6, DPYD.
|
||||
- CNV VCF: PGx CNV genes; bgzip+tabix; header updated to indicate "copy number of 5 may be reported";
|
||||
"Revised filter field delimiter to comply with VCF 4.3 specification".
|
||||
- Genotyping VCF: bgzip + tabix.
|
||||
|
||||
**Known issues (current page)**
|
||||
1. gtc-to-vcf locus combination: duplicate INDELs with HET calls on opposite strands (`D/I` + `I/D`) report `./.` instead of `0/1`.
|
||||
2. Corrupt/invalid GTC aborts the run instead of skipping.
|
||||
3. BPM vs CSV manifest mismatch does not abort gtc-to-vcf.
|
||||
4. SNV and indel designed to the same locus are never collapsed.
|
||||
5. Some indels missing from SNV VCF (designed indel ↔ reference mapping problem).
|
||||
6. Symbolic-link VCFs not supported as `star-allele call` input.
|
||||
7. Local Linux CLI and cloud do **not sort** `star_alleles.csv` and fields of `metabolizer_status.json`; Windows CLI does.
|
||||
8. New licence server `[ref-license-server]` returns "No valid licenses found." for v1.0 → keep `[ref-license-server-legacy]`.
|
||||
9. **GTC files do not support non-ASCII characters**: on non-English OS locales internal datetime fields may be written with
|
||||
non-ASCII characters, breaking later reads:
|
||||
```
|
||||
fail: ArrayAnalysis.CLI.App[0]
|
||||
System.IO.EndOfStreamException: Unable to read beyond the end of the stream.
|
||||
at System.IO.BinaryReader.ReadString()
|
||||
at ArrayAnalysis.Core.GtcFileLoader..ctor(String filePath) in /src/ArrayAnalysis.Core/GtcFileLoader.cs:line 161
|
||||
at ArrayAnalysis.Services.GtcFactory.CreateGtcFromSample(Sample sample, Boolean log)
|
||||
at ArrayAnalysis.Services.GtcToVcfService.Run(GtcToVcfInput input)
|
||||
at ArrayAnalysis.CLI.App.RunCliServiceAndReturnExitCode(BaseOptions opts) in /src/ArrayAnalysis.CLI/App.cs:line 110
|
||||
```
|
||||
Workaround: add `System.Globalization.Invariant: true` to `refcli.runtimeconfig.json`, regenerate GTCs with `genotype call`.
|
||||
|
||||
**Known issues present only in the archived (v1.0–v1.3 trees) copy of these notes, later removed/rewritten**
|
||||
- "For gtc-to-vcf, multi-allelic variants designed with multiple assays might not always collapse into one variant correctly
|
||||
and be reported as two separate variants instead."
|
||||
- "**Manifest names greater than 80 characters will cause failure when converting IDATs to GTCs.**" (silently dropped
|
||||
from the current notes — never listed as fixed).
|
||||
|
||||
**Known limitations**
|
||||
- PGx CNV + star-allele calling validated only for GDA_PGx_E2 product files.
|
||||
- Do not use `unsquash-duplicates` / `filter-loci` during gtc-to-vcf when star-allele calling is desired.
|
||||
- Cloud star-allele annotation: CPIC only (local: CPIC + DPWG).
|
||||
|
||||
### 1.2 Cloud Genotyping 1.0.0 — March 2024
|
||||
- Genotype + reports for human and **non-human** arrays in the cloud; configurable vendor-cloud interfaces.
|
||||
- New outputs: SNV VCF, Final Report, Locus Summary.
|
||||
- Known issue: "Some multi-nucleotide variant (MNV) designs reverse complement the 'Allele1/2 Top' fields in the Final Report"
|
||||
(still open in v1.4).
|
||||
- Limitation: "Genotyping only works on diploid organisms… Polyploid genotyping is not currently supported."
|
||||
|
||||
### 1.3 Cloud Methylation QC 1.0.0 — May 2024
|
||||
- Adjustable thresholds for **21 built-in controls**, detection p-value, proportion of probes passing, and **offset correction**;
|
||||
thresholds assign pass (1) / fail (0); failed metrics highlighted; failure localisation (bisulfite conversion, staining, hybridisation…).
|
||||
- Summary plots: p-value detection pass rate and PCA of beta values.
|
||||
- Per-CG detection p-value, beta and M values per sample.
|
||||
- Known issue: analysis may fail for **80–100 samples on large (>900K probe) arrays**; workaround: >100 samples.
|
||||
- Limitations: standard thresholds may not suit discontinued / semi-custom / custom chips or **NextSeq550 IDATs**;
|
||||
built-in controls may be absent on such chips.
|
||||
|
||||
### 1.4 Cloud Methylation QC 1.0.1 — February 2026
|
||||
- Hot-fix for the 80–100-sample / >900K-probe failure. Limitations unchanged.
|
||||
|
||||
### 1.5 v1.1.0 — September 2024
|
||||
|
||||
**Highlights**: EX PGx chips enabled; more PGx genes; custom `.egt` allowed in PGx; updated PGx databases; DPWG in cloud.
|
||||
|
||||
**New features / changes**
|
||||
- PGx products: GDA-ePGx, **GSAv4-ePGx (GSA-PGx-48v4-0_20079540_E2)**, **GCRA-ePGx (GCRA-PGx-24v1-0_20084467_C2)**;
|
||||
GDA-ePGx manifest moves to `GDA_PGx-8v1-0_20042614_G2` (archived notes contained this product/manifest/build table, build 38 only).
|
||||
- Star-allele annotation adds CYP2E1, CYP1A2, ABCG2, CYP2C8, HMGCR, UGT1A4, UGT2B15, F13A1, HLA-B*15:02.
|
||||
- CNV calling adds **SULT1A1** (features page: "6 target PGx genes across 9 target regions" → "7 genes across 10 regions").
|
||||
- "Extended bi-allelic PGx variants from source databases to multi-allelic variants based on the designs in the supported PGx products."
|
||||
- **GTCs from a custom cluster file may be used with the commercial CN model** (v1.0: hard error; v1.1+: warning
|
||||
`Cluster file ({GTC.egt}) is not the same as CN Model Cluster file ({CN_Model.egt}).`).
|
||||
- PGx database revision (PharmVar 6.0.5 → 6.1; PharmGKB snapshot 2023-08-30 → 2024-05-16; CPIC 1.30.0 → 1.38.0;
|
||||
DPWG June 2023; UGT nomenclature 2010-12-21; CYP legacy DB July 2024).
|
||||
- Star-allele JSON standardisation: `databaseSources` → `phenotypeDatabaseSources` + `starAlleleDatabaseSources`;
|
||||
`Phenotype` → `PhenotypeDatabaseAnnotation`; `missingVariants` + `allMissingVariants` → `missingVariantSites`;
|
||||
`supportingVariants` / `missingVariants` JSON-ised at gene and candidate-allele level; redundant info removed.
|
||||
- Updated tabix indexing "improving performance and disk usage for SNV VCF".
|
||||
- Resolved: symlink VCFs accepted by `star-allele call`; Linux/cloud sorting of `star_alleles.csv` / `metabolizer_status.json` fixed.
|
||||
- New CLI option `--query-license-quota`; `copy-number train --platform` accepts `EX` in addition to `LCG`.
|
||||
- Licensing: "In v1.1+, during analysis, precomputed quota is no longer checked" → over-quota runs fail only at the end with
|
||||
`Error updating usage. HTTP error status code: 409` and **results are not written to disk**.
|
||||
|
||||
**Known issues (v1.1)** — all v1.0 issues except #6/#7 plus:
|
||||
- REF/ALT delimited by `_` instead of `>` for some simple variants in `star_alleles.csv` / metabolizer JSON
|
||||
(e.g. `ryr1.38577931a_c`) — still open in v1.4.
|
||||
- "Occasional star-allele solution score discordance between Linux and Windows OS with concordant solution ranking."
|
||||
- "Rare intermittent memory issues during star allele calling. Example error message: `The model has been changed since
|
||||
the solution was last computed.`" — restart or more memory. (Fixed v1.4.)
|
||||
- Star-allele **annotation can fail mid-analysis when an allele is unknown** (observed: CYP2E1 mis-calls due to **cluster drift**). (Fixed v1.2.)
|
||||
- Licence server issue (#8) still present.
|
||||
|
||||
**Known limitations (v1.1)**
|
||||
- Only PharmVar/PharmGKB-defined alleles (no novel alleles).
|
||||
- CYP2D6 non-*36 exon-9-conversion alleles (e.g. *83) reported as *36 with *83 underlying (fixed v1.4).
|
||||
- Diploid only. Not backward compatible with v1.0 manifest (`…_E2` for v1.0, `…_G2` for v1.1).
|
||||
- VCFs must be gzipped + tabix-indexed (gtc-to-vcf default) for star-allele calling.
|
||||
|
||||
### 1.6 v1.2.0 — February 2025
|
||||
|
||||
**Highlights**: genome-wide CNV + LOH calling (VCF) for any human genotyping array; BAF bedGraph; ISCN/cytoband outputs.
|
||||
|
||||
**New features**
|
||||
- Cytogenetic calling: any human array ("detection abilities vary by array probe density and spacing"); CN up to **4+**;
|
||||
Phred-scaled quality; **mosaic tagging** of deletions/duplications; tested on GDA-Cyto-8, GSA-Cyto-24, CytoSNP-850K (iScan);
|
||||
adjustable minimum size / probe number for CNV and LOH.
|
||||
- BedGraph: new BAF bedGraph; extensions `BAF.bedgraph` / `LRR.bedgraph`; **`--smoothing` added to gtc-to-bedgraph** (LRR); IGV compatible.
|
||||
- Cyto annotation JSON: per-sample and per-event statistics, gene count + names (RefSeq), **ISCN 2020** and cytoband (Ensembl).
|
||||
- PGx: new root command `pgx` grouping copy-number and star-allele.
|
||||
- Fixed: `pgx star-allele annotate` failing on an unknown allele (CYP2E1), "an issue introduced in version 1.1".
|
||||
- New licence server `[ref-license-server]` supported.
|
||||
|
||||
**Known issues (v1.2)** — carried: MNV Top-allele RC; corrupt GTC abort; non-ASCII GTC; SNV/indel not collapsed; missing indels;
|
||||
BPM/CSV mismatch; `D/I`+`I/D` → `./.`; memory "model has been changed"; `_` delimiter; Linux/Windows score discordance. New:
|
||||
- If genotyping sex = unknown, **cyto caller assumes male** → sex-chromosome results wrong for females.
|
||||
- ISCN only for variants **> 1 kb**; no ISCN for LOH.
|
||||
- Centromere calls likely false positives.
|
||||
- `cyto annotate` logs `No credential is provided` (harmless).
|
||||
- `cyto call` logs `CytoPlatform` hard-coded to `LCG` regardless of product (no algorithmic effect).
|
||||
- Non-default `--smoothing` in gtc-to-bedgraph produces **wrong LRR values**; use default 0; option to be disabled.
|
||||
- `cyto call` may throw an **overflow error** "in very rare cases when no variants are detected in noisy or low-quality samples".
|
||||
- Min DEL/LOH/DUP thresholds shown in **wrong units** in annotation JSON when set above the calling thresholds.
|
||||
- Cyto variants with **QUAL 0** not passed into annotation JSON.
|
||||
- CYP2A6 *1 definition wrongly includes `NC_000019.10:g.40848264_40848265delinsT`.
|
||||
- Cloud cyto (and cyto + EMG) may fail at **~200 samples** per batch (memory).
|
||||
- ABCG2 reference-allele samples miss phenotype annotation.
|
||||
|
||||
**Known limitations (v1.2)**: diploid only; constitutional only (no oncology); validated only on 3 cyto chips; large events may be
|
||||
fragmented; **no mosaic fraction and no mosaic ISCN**; on `CytoSNP-850Kv1-4_iScan_B`, `GSACyto-24v1_20044998_C`,
|
||||
`GDACyto-8v1-0_20047166_E` manifests **no PAR calling/visualisation**; **GT hard-coded to 1/1 in cyto VCF**; NextSeq550 IDATs
|
||||
untested; PGx limitations as v1.1. Local-guide note (v1.2 only): "VCFs generated on Windows machines will not work when manually
|
||||
uploading to EMG. Only Linux-based VCFs will work." (fixed in cyto+EMG 1.3.0).
|
||||
|
||||
### 1.7 v1.2.0 + EMG v38 automatic case creation — June 2025
|
||||
- Automatic EMG case creation after "Cytogenetics analysis + EMG interpretation" completes (vendor cloud platform).
|
||||
- Limitation: analysis type offered regardless of EMG subscription/SNS setup; without them it silently runs as plain cyto analysis
|
||||
and no case is created. Cloud UI: "Bedgraph Smoothing window size" disabled.
|
||||
|
||||
### 1.8 v1.3.0 — August 2025
|
||||
|
||||
**Highlights**: mosaic fraction estimation; improved sex-chromosome calling incl. PAR; new QC metrics in cyto JSON.
|
||||
|
||||
**Genotyping & core**
|
||||
- GenomeStudio-compatible samplesheet; **separate IDAT and GTC samplesheets deprecated** (`--idat-sample-sheet`,
|
||||
`--gtc-sample-sheet` → `--sample-sheet`).
|
||||
- Samplesheet user columns propagated into `gt_sample_summary`.
|
||||
- **Samples failing IDAT→GTC are now listed** in gt_sample_summary: `Autosomal Call Rate` = `Call Rate` = 0;
|
||||
`Log R Ratio Std Dev` and `TGA_Ctrl_5716 Norm R` = `NaN`.
|
||||
- `--smoothing` **removed** from gtc-to-bedgraph.
|
||||
- New `genotype call --normalize-sample-name` (ignore IDAT sample name, use SentrixBarcode_Position) — present only in v1.3.
|
||||
|
||||
**Cytogenetics**
|
||||
- Fixed overflow crash on noisy samples; fixed ~200-sample memory limit.
|
||||
- **Length-normalised median copy number**: removed 1 kb lower limit on included segments (was ">1 kB segments aggregated";
|
||||
now "length-weighted copy numbers of each variant are aggregated").
|
||||
- Mosaic fraction reported; mosaic events above user-defined fraction can be **promoted** (`--max-mosaic-fraction`, HIGHFRACTION tag).
|
||||
- Reduced STDOUT verbosity of annotate; sample-level `medianLogRDev`; chromosome-level QC metrics; event-level `effectiveSize`.
|
||||
- **Variants filtered by effective size in `cyto call` but by raw size in `cyto annotate`.**
|
||||
- Fixed wrong-unit thresholds in JSON; fixed QUAL-0 CNVs missing from JSON.
|
||||
- Cyto caller now **attempts to resolve Unknown sex**; log "Sample XXX sex updated from Unknown to Male."
|
||||
- Fixed `CytoPlatform` log always `LCG`.
|
||||
|
||||
**PGx**: ABCG2 reference alleles annotated with default "Normal"; CYP2A6 *1 definition corrected.
|
||||
|
||||
**Known issues (v1.3)**: corrupt GTC abort; MNV Top RC; non-ASCII GTC; SNV/indel collapse; missing indels; BPM/CSV mismatch;
|
||||
`D/I`+`I/D` → `./.`; memory "model has been changed"; `_` delimiter; plus:
|
||||
- **Samplesheet empty columns**: `SentrixBarcode_A,SentrixPosition_A,,` → `System.ArgumentException : Duplicate column found. Column names
|
||||
are case-insensitive…`; a single trailing empty column → an empty `"": ""` field in gt_sample_summary.
|
||||
- **>1K-sample samplesheets significantly slow** analysis; split into ≤1K batches.
|
||||
|
||||
**Known limitations (v1.3)**: unsquash/filter-loci vs star-allele; diploid only; unresolved-unknown-sex → male (not logged);
|
||||
ISCN only >1 kb; centromere FPs; no ISCN for LOH; constitutional only; validated on 3 cyto chips (NextSeq550 processed but not
|
||||
validated); fragmented large events; GT hard-coded 1/1; **tabix index differs from `bcftools index --tbi`**
|
||||
(`bcftools index --stats/--nrecords` → "index … does not contain any count metadata"; re-indexing with htslib may break
|
||||
downstream `pgx star-allele call`); **Windows vs Linux `pgx star-allele call` differences on 1576 samples**:
|
||||
|
||||
| Field | # differences |
|
||||
|---|---|
|
||||
| Collapsed Star-Alleles | 2 |
|
||||
| Missing/Masked Core Variants | 1 |
|
||||
| Solution Long | 1 |
|
||||
| Supporting Variants | 2 |
|
||||
|
||||
(all overall solutions concordant); v1.3 incompatible with EMG v38 (cannot upload v1.3 cyto VCF; keep cyto+EMG 1.2.0);
|
||||
no novel alleles; CYP2D6 *83 reported as *36.
|
||||
|
||||
### 1.9 v1.3.0 + EMG V100.39.0 — October 2025
|
||||
- "Cytogenetics analysis + EMG interpretation 1.3.0" released; **VCFs generated on Windows now compatible with EMG**.
|
||||
- Same subscription/SNS limitation as 1.2.0.
|
||||
|
||||
### 1.10 v1.4.0 — May 2026
|
||||
|
||||
**Highlights**: QC Report (Control Dashboard replacing GenomeStudio's, Automated QC, Sample QC Heatmaps, Trend Analysis);
|
||||
cyto GAINLOH + better mosaic detection; PGx SV star-allele calling, no-call handling, redesigned outputs; improved MAV calling.
|
||||
|
||||
**Cytogenetics**
|
||||
- **GAINLOH** detection (gain + LOH).
|
||||
- Size/probe filters **per variant type** (DEL, DUP, LOH, GAINLOH, mosaic DEL/DUP/LOH).
|
||||
- Mosaic detection "at approximately **>15%** for mosaic states with a one-copy difference (CN2/CN1, CN2/CN3 diploid; CN1/CN0, CN1/CN2 haploid)".
|
||||
- **Mosaic duplication on male Y** supported.
|
||||
- CNV/LOH calls validated against expected LRR pattern ("most probes in an autosomal CN gain region should exhibit positive LRR");
|
||||
non-conforming variants filtered.
|
||||
- Fixed: first variant missing ISCN annotation in JSON.
|
||||
- **Removed the 5% mosaic-fraction hard cutoff** (sub-5% variants can be reported; `LowFraction` FILTER appears in VCF header).
|
||||
- **LOH regions < 500 kb flanked by copy-neutral regions removed** (false-positive reduction).
|
||||
|
||||
**QC Report**: sample- and dataset-level control/functional metrics; cloud single-dataset, local multi-dataset + cross-run;
|
||||
sample QC table + interactive HTML; thresholds via YAML + in-report configuration.
|
||||
|
||||
**PGx**
|
||||
- PGx CNV calling "reduce[d] sensitivity to outlier samples".
|
||||
- Targeted star-allele accuracy improvements; **structural-variant star-allele calling**.
|
||||
- Separation of "Per-sample variant no-calls" vs "Variants not represented on the array product", with affected alleles listed.
|
||||
- For star-allele no-calls: **ranked candidate diplotypes** with "variant delta annotations".
|
||||
- RSIDs + probe IDs in supporting and no-call variant outputs.
|
||||
- Genotype, metabolizer status and guideline reported directly in CSVs; CSV output split into multiple files.
|
||||
- Fixed memory issue ("model has been changed…"); fixed CYP2D6 exon-9 conversion (*83 no longer reported as *36).
|
||||
|
||||
**Genotyping & core**
|
||||
- Sample-name precedence (samplesheet `Sample_Name` > IDAT/GTC metadata > `SentrixBarcode_Position`).
|
||||
- Cloud genotyping outputs control-based QC metrics + QC report.
|
||||
- `--use-infI-nc-info` for MAV calling.
|
||||
|
||||
**Bug fixes**: samplesheet empty columns handled; **`D/I`+`I/D` opposite-strand duplicate INDEL HETs now `0/1`**.
|
||||
|
||||
**Important note**: self-contained **.NET 9** application; Linux needs native OS deps (.NET DEB/RPM dependency lists).
|
||||
|
||||
**Known issues (v1.4)**
|
||||
- PGx: **cluster-file mismatch between GTCs and PGx CNV model is now a fatal error** (was a warning in 1.1–1.3); fix = retrain CN model.
|
||||
- Windows vs Linux star-allele minor differences (as v1.3); "Occasional star-allele solution **unresolved variants** discordance between
|
||||
Linux and Windows OS with concordant solution ranking."
|
||||
- For star-allele no-calls, alternates may have **large variant edit distance ≥ 3** when low-quality variant calls are present.
|
||||
- `_` instead of `>` delimiter (still open).
|
||||
- **JSON candidate solutions include supporting details only for the first allele** of the diplotype (CSV complete).
|
||||
- Genotyping: corrupt GTC abort; MNV Top RC; non-ASCII GTC; SNV/indel collapse; missing indels; BPM/CSV mismatch silent;
|
||||
**`genotype call` / `qc call` with mismatched IDAT and manifest may silently produce inaccurate results** (watch low call rate,
|
||||
many NaN QC values); >1K samplesheet slowness.
|
||||
|
||||
**Known limitations (v1.4)**
|
||||
- Cyto: unresolved unknown sex → male (unlogged); ISCN only >1 kb; centromere FPs; "ISCN annotations are not fully supported for LOH
|
||||
and mosaic variants"; constitutional only; validated on GDA-Cyto-8, GSA-Cyto-24, CytoSNP-850K (**iScan and NextSeq550**);
|
||||
"assumes that most genomic regions are diploid… variants in **triploid samples** may be missed"; fragmented large events;
|
||||
GT hard-coded 1/1; no mosaic ISCN; **older cyto model files and cloud configurations incompatible with v1.4**.
|
||||
- QC report UI: hover text truncated at **40 chars**; heatmap button misalignment for many plates/barcodes; faceting groups >**6**
|
||||
levels into "Other"; after viewing >**12,000**-sample dataset cannot switch to one <**1,000** (reload); moving/resizing a selection
|
||||
can freeze the page.
|
||||
- PGx: unsquash/filter-loci + gzip/tabix requirements; no novel alleles.
|
||||
- Genotyping: diploid only; tabix ≠ bcftools index (as v1.3).
|
||||
|
||||
### 1.11 v1.4.0 + EMG V100.40.0 — June 2026
|
||||
- "Cytogenetics analysis + EMG interpretation 1.4.0" released; subscription/SNS limitation unchanged.
|
||||
|
||||
### 1.12 Resolved-issue ledger
|
||||
|
||||
| Issue | Introduced/first listed | Resolved |
|
||||
|---|---|---|
|
||||
| Symlink VCF input to star-allele call | 1.0 | 1.1 |
|
||||
| Linux/cloud not sorting star-allele outputs | 1.0 | 1.1 |
|
||||
| Cluster file must equal CN-model cluster file (error) | 1.0 | 1.1 (warning) → **re-hardened to error in 1.4** |
|
||||
| Annotate fails on unknown allele (CYP2E1 cluster drift) | 1.1 | 1.2 |
|
||||
| New licence server unsupported | 1.0 | 1.2 |
|
||||
| gtc-to-bedgraph `--smoothing` wrong LRR | 1.2 | 1.3 (option removed) |
|
||||
| cyto overflow crash with no variants | 1.2 | 1.3 |
|
||||
| cyto ~200-sample memory limit | 1.2 | 1.3 |
|
||||
| JSON threshold units wrong | 1.2 | 1.3 |
|
||||
| QUAL-0 cyto variants missing in JSON | 1.2 | 1.3 |
|
||||
| Unknown sex → male (now resolved when possible) | 1.2 | 1.3 (partial) |
|
||||
| `CytoPlatform` log = LCG | 1.2 | 1.3 |
|
||||
| ABCG2 reference allele no phenotype | 1.2 | 1.3 |
|
||||
| CYP2A6 *1 definition wrong | 1.2 | 1.3 |
|
||||
| Windows cyto VCF rejected by EMG | 1.2 | cyto+EMG 1.3.0 |
|
||||
| Samplesheet empty columns | 1.3 | 1.4 |
|
||||
| `D/I`+`I/D` → `./.` | 1.0 | 1.4 |
|
||||
| "Model has been changed…" memory issue | 1.1 | 1.4 |
|
||||
| CYP2D6 *83 reported as *36 | 1.1 | 1.4 |
|
||||
| First cyto variant missing ISCN | (unlisted) | 1.4 |
|
||||
| Methylation 80–100 samples, >900K probes | Meth 1.0.0 | Meth 1.0.1 |
|
||||
| Manifest name > 80 chars fails IDAT→GTC | 1.0 | never stated; silently dropped from notes |
|
||||
| Multi-assay MAV not collapsing | 1.0 | silently rewritten as "SNV and indel not collapsed" |
|
||||
|
||||
Never resolved (open in v1.4): corrupt GTC abort; non-ASCII GTC strings; MNV Top-allele reverse-complement in Final Report;
|
||||
SNV/indel same-locus not collapsed; unmapped indels; BPM/CSV mismatch not detected; `_` vs `>` delimiter; Linux/Windows PGx
|
||||
differences; tabix index lacking count metadata; >1K samplesheet slowness; GT hard-coded 1/1 in cyto VCF; unknown-sex default male;
|
||||
no novel star alleles; diploid only.
|
||||
|
||||
---
|
||||
|
||||
## 2. Version-diff tables (CLI, files, fields)
|
||||
|
||||
### 2.1 Command tree
|
||||
|
||||
| Command | v1.0 | v1.1 | v1.2 | v1.3 | v1.4 |
|
||||
|---|---|---|---|---|---|
|
||||
| `genotype call / gtc-to-vcf / gtc-to-bedgraph / help / version` | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `copy-number call/train/help/version` (root) | ✓ | ✓ | → `pgx copy-number …` | pgx | pgx |
|
||||
| `star-allele call/annotate/help/version` (root) | ✓ | ✓ | → `pgx star-allele …` | pgx | pgx |
|
||||
| `cyto call / annotate / help / version` | – | – | NEW | ✓ | ✓ |
|
||||
| `qc call / report / help / version` | – | – | – | – | NEW |
|
||||
| Syntax | `refcli [command] [req] [opt]` | same | `refcli [module] [sub-module (not needed for cyto)] [command] …` | + module table | + case-sensitivity note (`--threshold-callRate`, `--threshold-logRdev`) |
|
||||
|
||||
Doc slip: v1.0 lists "star-alle annotate" [sic]; v1.2 input table lists cyto database option as `--database` (real: `--annotation-db`, corrected v1.4).
|
||||
|
||||
### 2.2 Options per command
|
||||
|
||||
Common to all commands in all versions: `--debug` (stack traces, false), `--json-log` (false), `--help`, `--version`.
|
||||
|
||||
**genotype call**
|
||||
|
||||
| Option | v1.0 | v1.1 | v1.2 | v1.3 | v1.4 |
|
||||
|---|---|---|---|---|---|
|
||||
| `--bpm-manifest` (req) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `--cluster-file` (req) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `--idat-folder` | req, XOR with sheet, recursive | = | = | optional; overrides samplesheet `RootFolder` | = |
|
||||
| `--idat-sample-sheet` (CSV/JSON paths) | ✓ | ✓ | ✓ | REMOVED | – |
|
||||
| `--sample-sheet` (filter + metadata) | – | – | – | NEW | ✓ |
|
||||
| `--gencall-cutoff` | 0.15 | 0.15 | 0.15 | 0.15 | 0.15 |
|
||||
| `--num-threads` | ✓ (no default documented) | ✓ | ✓ | ✓ | ✓ |
|
||||
| `--output-folder` | "mirrors IDAT folder structure" | = | = | mirroring text removed | = |
|
||||
| `--normalize-sample-name` (false) | – | – | – | NEW | REMOVED (precedence rules) |
|
||||
|
||||
**genotype gtc-to-vcf**
|
||||
|
||||
| Option | v1.0 | v1.1 | v1.2 | v1.3 | v1.4 |
|
||||
|---|---|---|---|---|---|
|
||||
| `--bpm-manifest`, `--csv-manifest` (SourceSeq), `--genome-fasta-file` (.fai alongside) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `--gtc-folder` / `--gtc-sample-sheet` | req XOR | = | = | folder optional, `--sample-sheet` replaces gtc sheet | = |
|
||||
| `--auxiliary-loci` (VCF of MNV defs) | ✓ | + restrictions documented | ✓ | ✓ | ✓ |
|
||||
| `--disable-genome-cache` | ✓ | + explained | ✓ | ✓ | ✓ |
|
||||
| `--filter-loci` (text file of probe names) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `--unsquash-duplicates` (false) | ✓ | + explained | ✓ | ✓ | ✓ |
|
||||
| `--no-bgzip` (false) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `--use-infI-nc-info` (false) | – | – | – | – | NEW |
|
||||
| Input described as | "GTC" | "GTC (v5)" | = | = | = |
|
||||
|
||||
**genotype gtc-to-bedgraph**: v1.0 `--bpm-manifest`, gtc folder/sheet, `--output-folder` (LRR only) → v1.2 BAF+LRR, `--smoothing`
|
||||
(default 0, buggy) → v1.3 `--smoothing` removed, `--sample-sheet`.
|
||||
|
||||
**(pgx) copy-number call**: `--cn-model` (req .dat), gtc folder/sheet (→ `--sample-sheet` v1.3), `--no-bgzip`, `--output-folder`. Unchanged otherwise.
|
||||
|
||||
**(pgx) copy-number train**: `--bpm-manifest` (`.msk` must sit next to BPM with same base name), `--genome-fasta-file`, gtc folder/sheet
|
||||
(→ sample-sheet v1.3), `--platform` (v1.0 `LCG` only; v1.1+ `LCG` for GDA-ePGx, `EX` for GSAv4/GCRA-ePGx; value = CSV manifest
|
||||
`Assay Format` header), `--disable-genome-cache`, `--output-folder` (default CWD).
|
||||
|
||||
**(pgx) star-allele call**: `--database` (.zip), `--license-server-url` (req, `https://user:pass@host`), `--vcf-folder` (`*.snv.vcf.gz` + `*.cnv.vcf.gz`),
|
||||
`--output-folder`; **`--query-license-quota` NEW v1.1**.
|
||||
|
||||
**(pgx) star-allele annotate**: `--star-alleles` (**.csv in v1.0–v1.3 → .dat in v1.4**), `--guidelines` (CPIC default | DPWG), `--output-folder`.
|
||||
|
||||
**cyto call**
|
||||
|
||||
| Option | v1.2 | v1.3 | v1.4 |
|
||||
|---|---|---|---|
|
||||
| `--cn-model` (cyto model .dat) | ✓ | ✓ | ✓ |
|
||||
| gtc folder / `--gtc-sample-sheet` | ✓ | → `--sample-sheet` | ✓ |
|
||||
| `--no-bgzip`, `--output-folder` | ✓ | ✓ | ✓ |
|
||||
| `--min-cnv-probes` | 10 | 10 (VCF filter) | REMOVED → per-type |
|
||||
| `--min-cnv-size` (kb) | **0** | **20** (effective size) | REMOVED → per-type |
|
||||
| `--min-loh-probes` / `--min-loh-size` | 500 / 3000 | 500 / 3000 (effective size) | 500 / 3000 |
|
||||
| `--min-del-probes/size` | – | – | 10 / 25 kb |
|
||||
| `--min-dup-probes/size` | – | – | 10 / 50 kb |
|
||||
| `--min-gainloh-probes/size` | – | – | 300 / 500 kb |
|
||||
| `--min-mdel-probes/size` | – | – | 10 / 25 kb |
|
||||
| `--min-mdup-probes/size` | – | – | 10 / 50 kb |
|
||||
| `--min-mloh-probes/size` | – | – | 500 / 3000 kb |
|
||||
| `--max-mosaic-fraction` | – | NEW 1.0 | 1.0 |
|
||||
| `--smoothing` (LRR window, probes each side) | 5 | 5 | 5 |
|
||||
|
||||
**cyto annotate**
|
||||
|
||||
| Option | v1.2 | v1.3 | v1.4 |
|
||||
|---|---|---|---|
|
||||
| `--annotation-db` (req), `--vcf-folder` (req), `--output-folder` | ✓ | ✓ | ✓ |
|
||||
| `--min-del-probes` / `--min-del-size` | 10 / **0** | 10 / 0 | 10 / **25 kb** |
|
||||
| `--min-dup-probes` / `--min-dup-size` | 10 / **0** | 10 / 0 | 10 / **50 kb** |
|
||||
| `--min-loh-probes` / `--min-loh-size` | 500 / 3000 | = | = |
|
||||
| gainloh / mdel / mdup / mloh pairs | – | – | 300/500, 10/25, 10/50, 500/3000 |
|
||||
| `--min-qual` | 20 | 20 | 20 |
|
||||
|
||||
Notes added v1.4: values <10 for any CNV probe threshold silently reset to 10; thresholds larger than effective genome size filter all variants.
|
||||
|
||||
**qc call (v1.4)**: `--array-type` (genotyping|methylation, req), `--csv-manifest` (req), `--idat-folder`, `--sample-sheet`, `--output-folder`.
|
||||
**qc report (v1.4)**: `--data` (comma list or parent folder auto-discovery), `--config` (YAML), `--label` (count must match datasets),
|
||||
`--output-folder`, `--output-format` (xlsx default | csv), `--threshold-callRate` (0.98), `--threshold-logRdev` (0.20); CLI > YAML > built-in.
|
||||
|
||||
Cloud cyto thresholds (UI): v1.2/v1.3 CNV min size 0 kb / 10 probes, LOH 3000 kb / 500 probes, smoothing 5 (0–1000), all 0–250000;
|
||||
v1.4 per-type set equal to CLI defaults above; GTC and SNV VCF output default False.
|
||||
|
||||
### 2.3 Output files
|
||||
|
||||
| File | v1.0 | v1.1 | v1.2 | v1.3 | v1.4 |
|
||||
|---|---|---|---|---|---|
|
||||
| `*.gtc` (GTC v5) | ✓ | ✓ | + legacy-GTC note | + i18n note | ✓ |
|
||||
| `*.snv.vcf.gz` + `.tbi` (VCF 4.1) | ✓ | + ploidy / collapse rules doc | + FILTER=PASS note | + MAV/ID/flip/PLINK notes | + Inf-I no-call section |
|
||||
| PGx `*.cnv.vcf.gz` | "CNV VCF" | ✓ | renamed "PGx CNV VCF" | ✓ | ✓ |
|
||||
| Cyto `*.cnv.vcf.gz` | – | – | NEW "Cytogenetics CNV and LOH VCF" | renamed "Cytogenetics VCF" | + MF, MOSAIC, HIGHFRACTION, LowFraction |
|
||||
| `cyto.cnv.dat` (metadata in vcf-folder, needed by annotate) | – | – | NEW | ✓ | ✓ |
|
||||
| Cyto annotation JSON (per sample) | – | – | NEW | + chromosome/QC fields | + GAINLOH/mosaic thresholds, sampleName |
|
||||
| BedGraph | single LRR | = | `LRR.bedgraph` + `BAF.bedgraph` | = | = |
|
||||
| `star_alleles.csv` (intermediate, call→annotate) | ✓ | ✓ | ✓ | ✓ | **replaced by `.dat`** |
|
||||
| Star-allele JSON (`metabolizer_status.json`) | ✓ | restructured | ✓ | ✓ | restructured |
|
||||
| `diplotype_summary.csv`, `diplotype_annotations.csv`, `supporting_variants.csv`, `unresolved_variants.csv`, `variants_not_covered_on_array.csv` | – | – | – | – | NEW (annotate) |
|
||||
| `gt_sample_summary.csv/.json` | ✓ | ✓ | ✓ | + samplesheet fields, failed samples | + Sample Name precedence |
|
||||
| CN Summary, `cn_batch_summary.csv` | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `gtc_final_report.csv`, `locus_summary.csv` (cloud) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| `controls.raw_metrics.csv`, `controls.qc_metrics.csv` | – | – | – | – | NEW (genotype call + qc call) |
|
||||
| `<Product>_QC_Report_YYYY_MM_DD.html`, `<Product>_QC_table_YYYY_MM_DD.<xlsx|csv>` | – | – | – | – | NEW (qc report; cloud genotyping + cyto) |
|
||||
| Methylation outputs (cloud) | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
|
||||
### 2.4 Field-level changes
|
||||
|
||||
**PGx CNV VCF header**: v1.0/1.1 example lacks ploidy lines in doc; v1.2 `##CNVOverallPloidy`, `##CNVGCCorrect` → v1.3+ `##OverallPloidy`, `##GCCorrect`.
|
||||
FORMAT `CN` ("CN=5 indicates 5 or 5+"), `NR` (aggregated normalised intensity); INFO `CNVLEN, PROBE, END, SVTYPE`; FILTER `Q7`, `SampleQuality`.
|
||||
ID pattern `CNV:<gene/region>:chr<chr>:<start>:<end>`. SULT1A1 region changed v1.1: 16:28615068–28623382 → 16:28603587–28613544
|
||||
(doc example still shows old POS/END with new ID coordinates — internal inconsistency). GSTT1 on `22_KI270879v1_alt`.
|
||||
|
||||
**Cyto VCF header**: v1.2 `##CNVgenomicPloidy`, `##CNVGCCorrect` → v1.3 `##OverallPloidy`, `##GCCorrect`. Constant lines `##platform=cytoplatform`,
|
||||
`##product=GDACyto-8v1-0_A`, `##source=refcli X Cyto`. ALT `<DEL>`, `<DUP>` ("Copy number gain heterozygous region"), `<LOH>`. FORMAT
|
||||
`GT:CN:NR:LRD` (+ `MF` v1.4 doc). INFO `SVLEN, PROBE, END, LOHTYPE (AOH|CNLOH|GAINLOH)` (+ `MOSAIC`, `HIGHFRACTION` v1.4 doc).
|
||||
FILTER `Q10`, `SampleQuality` (+ `LowFraction` v1.4). IDs `DEL:chr1:…`, `AOH:…`, `MosaicDEL:19:…`. **v1.4 example POS = ID start − 1**
|
||||
(`1 109687841 DEL:chr1:109687842:…`) whereas v1.2/1.3 examples had POS = start (padding-base convention change or doc edit).
|
||||
GAINLOH example is emitted as `<LOH>` with `LOHTYPE=AOH` and CN=3 (doc inconsistency). v1.4 adds QUAL formula `QUAL = -10*log10(1 - P(called state))`, cap 60, min 0.
|
||||
|
||||
**SNV VCF**: FORMAT `GT:GS:BAF:LRR`; GS header "For merged multi-assay or multi-allelic records, min GenCall score is reported"; QUAL `.`,
|
||||
FILTER always `PASS` (documented v1.2), INFO `.`; MAV records carry only GT and GS (documented v1.4).
|
||||
|
||||
**Star-allele CSV → diplotype_summary.csv**: v1.0–1.3 columns `Sample, Rank, Gene or Variant, Type (Haplotype|Variant), Solution, Solution Long,
|
||||
Supporting Variants, Missing/Masked Core Variants, All Missing Variants in Array, Collapsed Star-Alleles, Score, Raw Score:, Copy Number Solution`
|
||||
("Gene Region: Copy Number"). v1.4: `Sample, Solution Id, Gene or Variant, Type, Solution, Array-Equivalent Alleles (Allele 1), (Allele 2),
|
||||
No Call Variants, Alleles Impacted by No-Call, Background Alleles, Variant Change, Score, Raw Score, Copy Number Solution` ("GeneRegion=CopyNumber").
|
||||
Renames: Rank→Solution Id; Collapsed Star-Alleles→Array-Equivalent Alleles (split per allele); Missing/Masked→No Call Variants + Alleles Impacted;
|
||||
All Missing Variants in Array→variants_not_covered_on_array.csv; Supporting Variants→supporting_variants.csv
|
||||
(`Sample, Solution Id, Gene or Variant, Allele, HGVS, RSID, Chromosome, Position, Ref, Alts, Genotype, Variant Copy Number, GS, BAF, ProbeID`);
|
||||
unresolved_variants.csv (`Sample, Gene or Variant, Solution Id, Allele, HGVS, RSID, Reason (Not on Array|No Call), ProbeID`);
|
||||
diplotype_annotations.csv (`Sample, Gene, PGx Guideline, Genotype, Activity Score, Phenotype Database Annotation`); v1.3: multiple top solutions
|
||||
separated by `;` in `genotype`.
|
||||
|
||||
**Star-allele JSON**: v1.0 top: `softwareVersion, genomeBuild ("hg38"), databaseSources (string), mappingFile, pgxGuideline, sampleId, locusAnnotations[]`;
|
||||
locus: `gene, callType, genotype, activityScore, phenotype, qualityScore, rawScore, supportingVariants (string "Complete: *1 ( )"), candidateSolutions[
|
||||
rank, genotype, activityScore, phenotype, qualityScore, rawScore, alleles[solutionLong, supportingVariants, missingVariants, collapsedAlleles],
|
||||
copyNumberRegions, copyNumberSolution], allMissingVariants, allelesTested`.
|
||||
v1.1: `genomeBuild "38"`; `starAlleleDatabaseSources[]`, `phenotypeDatabaseSources[]`; `phenotypeDatabaseAnnotation` (e.g. "CYP2C9 Normal Metabolizer");
|
||||
supportingVariants objects `{chrom,pos,ref,alt,gt,gs,baf,id,alleleIds}` (all strings; ids sometimes with a leading space);
|
||||
`missingVariantSites[{id,alleleIds}]`; quality scores are strings at locus level, numbers inside candidates; `collapsedAlleles` e.g. `"*2.001"`.
|
||||
v1.4: locus adds `arrayEquivalentAlleles[{alleleId, ArrayEquivalentAlleles}]`, `variantNoCalls`, `allelesImpactedByNoCalls`,
|
||||
`variantsNotCoveredOnArray[{hgvs,rsid,alleleIds}]` (replaces missingVariantSites); candidate alleles `{alleleName, supportingVariants, unresolvedVariantSites,
|
||||
arrayEquivalentAlleles, backgroundAllele}` (solutionLong/collapsedAlleles/missingVariantSites removed), candidate adds `variantNoCalls,
|
||||
allelesImpactedByNoCalls, variantChange`; supportingVariants `{CN, chrom, pos, ref, alts, gt, gs, baf, hgvs, rsid, alleleIds}` — the doc example
|
||||
literally has keys `"CN:"`, `"chrom:"`, `"pos:"`, `"ref:"` (trailing colon; serialisation bug or doc artefact). Example genotype
|
||||
`*27/*36x2+*83+*10`, "CYP2D6 Indeterminate", `copyNumberRegions "p5, intron.2, exon.9"`, `copyNumberSolution "5, 5, 2"`.
|
||||
allelesTested list order changed (no longer numeric).
|
||||
|
||||
**Cyto annotation JSON**: v1.2 top: `annotateDb, softwareVersion, referenceGenome, annotationType (0=Constitutional,1=Oncology), genomeBuild (hg19/hg38),
|
||||
databaseSources, iscnVersion, sampleId, gcCorrect, minDelProbes, minDupProbes, minLOHProbes, minDelSize, minDupSize, minLOHSize, minQual,
|
||||
overallPloidy, callRate, logRDev, bafDev{AA,AB,BB}, numLOHOver1M, numLOHOver8M, totalSizeLOHOver1M, copyNumberMedian, percentLOH (string "1.59%"),
|
||||
sexEstimate ("Female"), traditionalNomenclature, microarrayNomenclature, chromosomeAnnotations[{id,numberDel,numberDup,numberLOH,numberMosaic}],
|
||||
locusAnnotations[{id,chrom,start (0-based: 120311441 for id …:120311442),end,callType,mosaicState,copyNumber,qualityScore,size,probeCount,percentHet,
|
||||
lrrMedian,lrrDev,bafDev,startCytoBand,endCytoBand,traditionalNomenclature,microarrayNomenclature,geneCount,genes[]}]`.
|
||||
v1.3: `annotationType` → string `"Constitutional"` (table still says integer); `medianLogRDev`; chromosome fields `size, percentHet, hasMosaicism,
|
||||
lrrMedian, lrrDev, numLOHOver1M, numLOHOver8M, totalSizeLOHOver1M, percentLOH, copyNumberMedian, copyNumberMean, minLogRRatio, maxLogRRatio,
|
||||
medianMosaicFraction ("." when none)`; locus `mosaicFraction` ("."), `effectiveSize`.
|
||||
v1.4: `sampleName`; `minGainlohSize, minMdelSize, minMdupSize, minMlohSize, minGainlohProbes, minMdelProbes, minMdupProbes, minMlohProbes`;
|
||||
chromosome `lrrMean`, `numberGainLOH`; sample-level LOH counts **include** GAINLOH, chromosome-level counts **exclude** GAINLOH.
|
||||
Example min sizes: v1.2 "50kb/50kb" (while CLI default was 0), v1.3 "20kb/20kb", v1.4 "25kb/50kb".
|
||||
|
||||
**gt_sample_summary**: v1.0 `Sample ID, Sample Name, Sample Folder, Autosomal Call Rate, Call Rate, Log R Ratio Std Dev, Sex Estimate,
|
||||
TGA_Ctrl_5716 Norm R` (last only on PGx products; <1 ⇒ TGA failure). v1.3 + user samplesheet columns appended verbatim; failed samples rows.
|
||||
v1.4 Sample Name from `Sample_Name`; reserved keywords (e.g. `Sex Estimate`) rejected as samplesheet columns.
|
||||
|
||||
**Warning/error catalogue**
|
||||
- v1.0: Reference allele not queried; Skipping non-mapped / intensity-only locus / indel; Failed to process entry; Incomplete match of source
|
||||
sequence; "Failed to combine genotypes due to ambiguity - exm1068284 (InfiniumII): TT, …"; "{n} sample(s) passed QC… Requires at least {m}";
|
||||
Invalid manifest path; Failed to load cluster file.
|
||||
- v1.1 +: "Failed to normalize and gencall sample: {sample_id}, it will be skipped. Error: The given key '{loci_id}' was not present in the
|
||||
dictionary." (BPM/EGT mismatch — sample skipped, not fatal); cluster-file vs CN-model warning; cluster load error cause "Corrupted file or unsupported version".
|
||||
- v1.3 +: `System.IO.EndOfStreamException` entry.
|
||||
- v1.4: "…due to ambiguity" renamed "…due to **inconsistencies**"; new "Failed to combine genotypes **all were No Calls**".
|
||||
|
||||
### 2.5 Input-file changes
|
||||
- v1.1: product files only from support site (MyIllumina custom manifests for design only); CSV manifest mapping rules (1-based; **PAR positions
|
||||
reported on X**; insertion = base 5′ of insertion on + strand; deletion = most 5′ deleted base); **Mask file (.msk)** documented (same base name
|
||||
as BPM); FASTA contig names must match manifest (`>1` not `>chr1`); NextSeq550 "for select arrays".
|
||||
- v1.2: PGx CN model vs **Cytogenetics model file**; "CN model file needs to be updated upon manifest revisions… mismatch … will cause an error
|
||||
during `pgx copy-number call` and `cyto call`"; Cytogenetics database (.zip, Ensembl + RefSeq, product-independent).
|
||||
- v1.3: unified **Sample Sheet** (CSV only; `SentrixBarcode_A, SentrixPosition_A` local / `beadChipName, sampleSectionName` cloud; optional
|
||||
`[Header]` with `RootFolder`, `[Data]`; `[Manifests]` ignored; CLI folder beats RootFolder; column names case-insensitive); JSON sheets dropped;
|
||||
PGx DB described as many-probes-to-one-HGVS map cross-referenced to SNV VCF IDs at runtime; FASTA only needed locally; cyto+EMG sheet
|
||||
`demographicSex` (M/F/U).
|
||||
- v1.4: CN/cyto model files named by manifest revision (e.g. B1) and must pair with that revision; cyto model "primarily contains probe GC content
|
||||
information", needs no retraining for a different cluster file; for custom products supply BPM + EGT + IDAT/GTC of ≥1 sample; samplesheet
|
||||
fields ≤ **500 chars**; Excel scientific-notation barcodes break IDAT lookup; Sample Name Determination; YAML config (null disables a check);
|
||||
cyto+EMG sheet keys switched to `SentrixBarcode_A,SentrixPosition_A,demographicSex`.
|
||||
|
||||
---
|
||||
|
||||
## 3. Internal hints (implementation, libraries, formats)
|
||||
|
||||
- **Runtime/stack**: .NET (self-contained; **.NET 9 from v1.4**), config `refcli.runtimeconfig.json`, `System.Globalization.Invariant` switch.
|
||||
Namespaces/classes: `ArrayAnalysis.CLI.App` (`RunCliServiceAndReturnExitCode(BaseOptions opts)`, App.cs:110 — verb/options pattern typical of
|
||||
CommandLineParser), `ArrayAnalysis.Core.GtcFileLoader` (ctor reads strings via `System.IO.BinaryReader.ReadString()` at line 161 → GTC strings are
|
||||
.NET 7-bit-length-prefixed; culture-formatted datetime written into GTC), `ArrayAnalysis.Services.GtcFactory.CreateGtcFromSample(Sample, Boolean log)`,
|
||||
`ArrayAnalysis.Services.GtcToVcfService.Run(GtcToVcfInput)`. Log format `fail: ArrayAnalysis.CLI.App[0]` = Microsoft.Extensions.Logging console
|
||||
formatter. Samplesheet validation throws `System.ArgumentException`. Errors like "The given key '…' was not present in the dictionary" =
|
||||
.NET `KeyNotFoundException` leaking from locus lookup (BPM↔EGT keyed by locus name).
|
||||
- **"The model has been changed since the solution was last computed."** — wording matches Google OR-Tools `MPSolver` (linear/integer
|
||||
programming) → the star-allele caller very likely solves an ILP/MIP per gene (Aldy-style). Linux/Windows score differences are consistent with
|
||||
solver/floating-point nondeterminism. (Inference.)
|
||||
- Star-allele quality: "negative log likelihood" for alternatives; `rawScore` excludes population-frequency prior; `qualityScore` includes it;
|
||||
alternatives kept if score ≥ 50% of best; "variant edit distance" used to rank no-call candidates.
|
||||
- Build metadata: `softwareVersion "refcli 1.1.0+9f82ed31d8c17e42b80f67a3e2b271f1a873e1d1"` (SemVer + git SHA); PGx DB names
|
||||
`GDA_ePGx_E2_DAv1.0.0.zip`, `<PRODUCT>-549-fix-annotate-sha.e56e884e….zip` (Jira key <PRODUCT>-549), `DAv1.3.0-rc3.zip`,
|
||||
`DAv1.4.0-rc2-pgx-mapping-sha.b164b4aa….zip`; `CytoAnnotateData_DAv1.2.0.zip`; `gda_mapping_53e0931.zip` (v1.0).
|
||||
- Licensing: `[ref-license-server-legacy]` (v1.0–1.1; legacy host from the product's original developer) → `[ref-license-server]` (v1.2+); per-sample quota; HTTP 409 on
|
||||
usage update when over quota; results discarded.
|
||||
- **Platform codes**: `LCG` (GDA / Infinium LCG chemistry) and `EX` (Infinium EX) = CSV manifest `Assay Format`; cyto log `CytoPlatform`; cyto VCF
|
||||
`##platform=cytoplatform`.
|
||||
- CN model stores the cluster-file identity (compares `{GTC.egt}` with `{CN_Model.egt}`) → GTC records the cluster-file name; CN/cyto models are
|
||||
keyed to manifest revision; PGx CNV model trained with `.msk` mask; cyto model = per-probe GC content (+ presumably reference LRR stats).
|
||||
- **GTC**: v5; "legacy GTCs" from AutoConvert/Beeline work for gtc-to-vcf but **not for cyto/PGx** → the product's own GTCs carry extra content or
|
||||
tighter conventions required downstream. BeadArrayFiles cited as parser.
|
||||
- **VCF/index**: claims VCF 4.1 but filter delimiter made VCF-4.3-compliant; own tabix writer (no n_no_coor / count metadata pseudo-bin);
|
||||
bgzip. GenomeStudio parity everywhere (samplesheet, Final Report, Locus Summary, Control Dashboard).
|
||||
- **Cyto algorithm hints**: GC correction flag, overall ploidy (length-weighted), LRR smoothing window (probes per side, default 5), Phred QUAL
|
||||
from state posterior, effective size (gaps >150× median probe spacing replaced by median spacing), LRR-pattern consistency filter, short-LOH
|
||||
(<500 kb, copy-neutral flanks) removal, mosaic fraction (NaN if too few informative probes), promotion above max fraction, sex resolution,
|
||||
whole-chromosome-only mode when LogRDev > 0.3, CN capped at 4+, GT always 1/1, annotationType Constitutional/Oncology, ISCN 2020,
|
||||
RefSeq `GCF_000001405.40-RS_2023_10`, Ensembl 112, genes = HGNC symbols + unmatched ENSG accessions.
|
||||
- **Methylation QC internals**: R/Bioconductor **SeSAMe** — logs `run_sesame_gs.log`, `convert_manifest_ilmn_sesame.log`, "Manifest in Sesame format
|
||||
(probes.csv)", **noob** background normalisation, **dye-bias correction**, detection p-value (0.05), BACR (BeadArray Controls Reporter) metrics
|
||||
(21 controls), background offset x = 3000, PCA of betas, `write_samplesheet.log`, `format_samplesheet.log`, `qc.log`.
|
||||
- **QC report**: Plotly-based self-contained HTML; YAML thresholds; control formulas (bkg_grn = max green of Extension A/T, bkg_red = max red of
|
||||
Extension C/G, x = 3000); IDAT metadata extracted: ImagingDate, SamplePlate, SampleWell, ScannerId, ScannerVersion.
|
||||
- **Cloud**: the vendor cloud platform, cloud data layer (renamed twice), AWS SNS "Analysis Success" event, payload v4, iCredits → BICs.
|
||||
|
||||
---
|
||||
|
||||
## 4. Numeric thresholds, defaults and limits (with changes)
|
||||
|
||||
| Item | Value(s) / change |
|
||||
|---|---|
|
||||
| GenCall no-call cutoff | 0.15 (all versions) |
|
||||
| Min RAM | 16 GB (v1.0–1.1) → "16 GB available" (v1.2–1.3) → **32 GB available (v1.4)**; 8 cores; 30 GB disk; cyto annotate ≥5 GB free |
|
||||
| PGx CNV batch | min 24 samples, 22 passing LogRDev < 0.2; recommended ≥96 passing LogRDev ≤ 0.2; ≤500 GDA-ePGx samples/batch on std HW; samples from ≤2-week window may be pooled |
|
||||
| PGx CN train | min 96 samples, 90 passing LogRDev ≤ 0.2; recommend ≥150; diminishing returns >3000; exclude LogRDev > 0.2, call rate < 0.99, TGA ctrl < 1.0 |
|
||||
| PGx CNV VCF | QUAL Phred cap 60, min 0; FILTER Q7 (<7); SampleQuality LogRDev > 0.2; CN=5 means ≥5 |
|
||||
| PGx coverage | CNV: 6 genes / 9 regions (v1.0) → 7 / 10 (v1.1, +SULT1A1) |
|
||||
| Star-allele | in-silico call rate ≥ 99%, diplotype accuracy ≥ 90% (v1.0); alt solutions listed if ≥50% of top score; review supporting variants with GS < 0.15; no-call alternates may have edit distance ≥ 3 |
|
||||
| Star-allele DB versions | PharmVar 6.0.5→6.1; PharmGKB 2023-08-30→2024-05-16; CPIC 1.30.0→1.38.0; DPWG June 2023 |
|
||||
| Linux/Windows PGx discordance | 1576 samples: 2 / 1 / 1 / 2 field differences (v1.3) |
|
||||
| Cyto VCF | QUAL cap 60, Q10 (<10); SampleQuality LogRDev > 0.2; LogRDev > 0.3 ⇒ whole-chromosome events only; CN max 4+ |
|
||||
| Cyto CNV size (call) | v1.2 min CNV 0 kb/10 probes → v1.3 **20 kb (effective)**/10 → v1.4 DEL 25 kb/10, DUP 50 kb/10 |
|
||||
| Cyto annotate DEL/DUP size | 0 kb (v1.2–1.3) → 25 / 50 kb (v1.4) |
|
||||
| LOH | 3000 kb / 500 probes (all); GAINLOH 500 kb / 300 probes (v1.4); mosaic DEL/DUP/LOH = non-mosaic values |
|
||||
| min-qual (annotate) | 20 |
|
||||
| Probe threshold floor | values < 10 reset to 10 (v1.4 doc) |
|
||||
| Smoothing | cyto 5; bedgraph 0 (v1.2 only) |
|
||||
| Mosaic | v1.3: "<20% may be missed, <5% not called"; v1.4: detect ≳15% for one-copy states, "<15–20% may be missed", 5% cutoff removed; max-mosaic-fraction 1.0 |
|
||||
| LOH short-segment filter | <500 kb with copy-neutral flanks removed (v1.4) |
|
||||
| ISCN | only events > 1 kb |
|
||||
| copyNumberMedian | v1.2 segments >1 kb only → v1.3 all segments length-weighted |
|
||||
| Effective size | gaps > 150 × median probe spacing |
|
||||
| Visual-inspection flag | >10 events (DEL/DUP/AOH) per chromosome |
|
||||
| Cyto cloud memory | fails ~200 samples (v1.2), fixed v1.3 |
|
||||
| Batch limits (cloud) | genotyping 1152, PGx 384, cyto 1152, methylation 1152 |
|
||||
| Samplesheet | slow >1K samples; fields/columns ≤500 chars (v1.4) |
|
||||
| Manifest name | >80 chars failed IDAT→GTC (v1.0-era) |
|
||||
| Final report | >1M loci × >100 samples ⇒ 50+ GB |
|
||||
| QC report | callRate 0.98, logRDev 0.20 default; others unset; UI: 40-char hover, >6 facet levels → "Other", 12,000/1,000-sample switch bug |
|
||||
| Methylation | 21 controls; offset x = 3000; p-value 0.05; failure at 80–100 samples with >900K probes (fixed 1.0.1); FFPE restoration threshold 0 → 1 if restore kit used |
|
||||
| Hyb control conc. | 5 pM / 1 pM / 0.2 pM |
|
||||
|
||||
---
|
||||
|
||||
## 5. Gap analysis vs SimurgArray plan — behaviours to implement or test
|
||||
|
||||
Wording is generic ("the reference tool"). Each item names the plan section it touches.
|
||||
|
||||
### 5.1 File I/O and robustness (§6, §14, §16)
|
||||
1. **GTC string encoding** (§6 GTC writer): write all strings as length-prefixed UTF-8/ASCII with 7-bit varint length; format timestamps with
|
||||
invariant culture and ASCII digits only; add a fuzz/golden test with a non-English locale (e.g. `LC_ALL=tr_TR`, `ar_SA`, `ja_JP`) and verify the GTC
|
||||
still round-trips and is readable by the oracle parser. Reader must survive files written by the reference tool under the same bug (clean error, not crash).
|
||||
2. **Corrupt GTC / IDAT handling**: skip-and-report per sample (exit code 3) rather than abort; include failed samples in the sample summary with
|
||||
call rate 0 and LogRDev/NaN fields, matching the reference behaviour since v1.3.
|
||||
3. **Manifest/cluster/IDAT consistency checks** the reference lacks: BPM vs CSV (locus names, count, order), BPM vs EGT (missing key → reference
|
||||
skips sample with a dictionary error), IDAT vs BPM (address coverage; warn on low call rate / many NaN QC). Make these explicit validation errors (exit 2).
|
||||
4. **Long names**: test manifest/cluster names > 80 chars, paths with spaces and non-ASCII; cluster file names limited to `[A-Za-z0-9_-]` by the reference — we should accept anything.
|
||||
5. **Duplicate IDAT pairs** for the same barcode_position under one input tree: the reference picks one nondeterministically (OS-dependent). Detect and fail (or deterministic rule + warning).
|
||||
6. **Overwrite policy**: the reference silently overwrites outputs; the plan's `--force` rule is stricter — keep it and test it.
|
||||
7. **Batch scale**: samplesheets > 1000 samples must not degrade super-linearly; add a 10k-row samplesheet benchmark (§19).
|
||||
|
||||
### 5.2 Samplesheet semantics (§5, §14)
|
||||
8. Accept GenomeStudio sheets: optional `[Header]`/`[Data]`, ignore `[Manifests]`, `RootFolder` overridden by CLI folder; keys `SentrixBarcode_A`,
|
||||
`SentrixPosition_A` (and cloud-style `beadChipName`, `sampleSectionName`).
|
||||
9. Column names case-insensitive; duplicate (case-folded) names → clear error; empty column headers tolerated (ignored); reserved output names
|
||||
(`Sex Estimate`, etc.) rejected or namespaced; values/columns length cap; barcodes in scientific notation/decimal (`2.04753E+11`, `204753010023.0`) detected with a helpful error.
|
||||
10. **Sample-name precedence**: sheet `Sample_Name` (must be non-empty for all rows and must not collide with another sample's barcode_position ID)
|
||||
> name embedded in IDAT/GTC > `barcode_position`. Never rewrite the name inside the GTC; downstream outputs follow the sheet of that command.
|
||||
11. Pass-through of arbitrary user columns into the sample summary (CSV + JSON), preserving order.
|
||||
|
||||
### 5.3 Genotyping and VCF export (§7, §8)
|
||||
12. **Multi-assay / multi-allelic combination** (§8.3): enumerate genotypes over queried alleles; filter by each assay's call respecting Infinium II
|
||||
A/T–C/G blindness; 0 survivors = "inconsistent", >1 = "ambiguous", all NC = "all no calls" — emit distinct warnings. Test the two published
|
||||
scenarios: Inf II [A/G]=AA + Inf I [T/G]=TT → NC (ambiguity AT/TT); Inf I [T/A]=AA + Inf II [A/G]=NC → NC.
|
||||
13. Optional **"Infinium I no-call means missing signal"** mode (reference `--use-infI-nc-info`): an Inf-I NC excludes its allele from candidates;
|
||||
document the higher FP risk and that clusters for zeroed-out MAV assays must exist in the cluster file.
|
||||
14. **Opposite-strand duplicate indels** (`D/I` + `I/D` both HET) must yield `0/1` (reference bug until v1.4) — add a regression test.
|
||||
15. **SNV + indel at the same position** are not merged by the reference; decide policy (keep separate by default for compatibility; optional merge) and test.
|
||||
16. MNV via auxiliary-loci VCF: must be MNV, not multi-allelic, not duplicated; test each restriction. Also test Final-Report TOP alleles for MNV
|
||||
designs (reference reverse-complements some — do not replicate).
|
||||
17. Merged/MAV records: GS = min of assays; **no BAF/LRR**; IDs joined with `,` (SNP) / `;` (indel); FILTER constant `PASS`, QUAL `.`.
|
||||
18. Unmapped loci (`Chr=0`), intensity-only probes, unmappable indels → skipped with specific warnings; count them in the run summary.
|
||||
19. Indel coordinate conventions from manifest (insertion = base 5′ of insertion on +; deletion = most 5′ deleted base; 1-based; PAR mapped to X)
|
||||
and REF/ALT "flip" where the manifest calls an insertion a deletion — test against dbSNP-normalised truth.
|
||||
20. Ploidy: haploid GT on Y, MT and non-PAR X for males; unknown-sex policy must be explicit.
|
||||
21. BAF/LRR orientation relative to REF (1−BAF vs manifest orientation) — already in plan; add a test on a locus where A allele = ALT.
|
||||
22. Contig naming: FASTA contigs must match manifest (`1` vs `chr1`); plan's alias table should be tested with both spellings and `22_KI270879v1_alt`.
|
||||
23. **Index files**: write tabix/CSI via htslib so `bcftools index --stats/--nrecords` works (reference index lacks count metadata); verify downstream readers.
|
||||
24. VCF version header vs content (reference claims 4.1 but uses 4.3 filter delimiters) — choose one and validate with `bcftools`/`vcf-validator`.
|
||||
25. `--unsquash-duplicates` must not unsquash multi-allelic duplicate designs (reference rule); test.
|
||||
26. Sample-summary field `TGA_Ctrl_5716 Norm R`-style "named control probe normalised R" should be supported generically (configurable probe list), omitted when absent.
|
||||
|
||||
### 5.4 Cross-platform determinism (§15, U11)
|
||||
27. The reference produces different star-allele scores/fields and different output sorting on Windows vs Linux, and picks different duplicate
|
||||
IDATs per OS. SimurgArray must: sort every output deterministically (sample, gene, rank, then lexical tie-breakers), avoid hash-order iteration,
|
||||
fix solver seeds/tolerances, and run golden tests on x86 and ARM (U11).
|
||||
|
||||
### 5.5 BedGraph / visualisation (§13)
|
||||
28. BAF and LRR bedGraphs; if any LRR smoothing is offered, it must never change the stored raw values (reference smoothing corrupted LRR) — golden test: smoothing=0 equals raw.
|
||||
|
||||
### 5.6 Copy number, LOH, mosaicism (§10, v2)
|
||||
29. Defaults to mirror (latest reference): DEL 25 kb/10 probes, DUP 50 kb/10, LOH 3000 kb/500, GAINLOH 500 kb/300, mosaic same as germline,
|
||||
min QUAL 20 (report) with Q10 FILTER, QUAL = −10·log10(1−P) capped 60, min probes floored at 10, thresholds > genome ⇒ empty output (not error).
|
||||
30. **Effective size** (gaps > 150× median spacing collapsed) used for filtering at call time; decide and document whether annotation filters use raw or
|
||||
effective size (reference differs between steps) — add a test with a variant spanning a centromere gap.
|
||||
31. States: DEL, DUP (CN up to 4+ capped), LOH subtypes AOH / CN-LOH / **GAINLOH** (CN=3 with LOH), mosaic DEL/DUP/LOH incl. **mosaic Y duplication in males**;
|
||||
haploid-region mosaic states (CN1/CN0, CN1/CN2).
|
||||
32. Mosaic fraction: NaN/"." when insufficient informative probes; `LowFraction` filter below a threshold instead of hard drop; promotion when
|
||||
MF ≥ max-fraction (`HIGHFRACTION`) keeping MF; target sensitivity ≳15% for one-copy states; test 5%, 10%, 15%, 20% simulated mixtures.
|
||||
33. Post-filters: LRR-direction consistency (gain ⇒ mostly positive LRR, loss ⇒ negative); remove LOH < 500 kb flanked by copy-neutral segments;
|
||||
LogRDev > 0.2 ⇒ SampleQuality flag; LogRDev > 0.3 ⇒ whole-chromosome events only; >10 events/chromosome ⇒ review flag.
|
||||
34. **Sex handling**: resolve unknown sex inside the CN caller (log when changed); never silently default to male — emit an explicit flag. PAR
|
||||
must be callable (reference failed on some manifests).
|
||||
35. Overflow/degenerate cases: zero variants in a noisy sample, all-NaN LRR chromosome, single-probe chromosomes, QUAL 0 events (must still reach JSON),
|
||||
very large batches (≥200 samples memory), triploid genomes (document limitation), centromere artefact masking.
|
||||
36. VCF genotype: reference hard-codes GT=1/1 for all CN events; SimurgArray should emit a meaningful GT (e.g. `0/1` for het DEL, `./.` or
|
||||
CN-only for symbolic SVs) with an optional compatibility mode; fix POS convention (symbolic SV POS = base before event per VCF spec) and test
|
||||
that ID coordinates, POS and END agree.
|
||||
37. Annotation JSON parity fields worth reproducing: overall ploidy, callRate, logRDev, medianLogRDev, bafDev per genotype, LOH counts >1 Mb/>8 Mb,
|
||||
percentLOH, length-weighted CN median (all segments, no 1 kb floor), per-chromosome stats (lrrMean/median/dev, min/max LRR, CN mean/median,
|
||||
percentHet, hasMosaicism, medianMosaicFraction, counts incl./excl. GAINLOH), per-event effectiveSize, ISCN 2020 (events > 1 kb; also for
|
||||
LOH and mosaic, which the reference lacks; ensure the first event is annotated), cytobands, genes (HGNC + unmatched Ensembl IDs).
|
||||
38. Units: always store sizes in bp internally and render units consistently (reference printed thresholds in wrong units).
|
||||
|
||||
### 5.7 Pharmacogenomics (§11, v2)
|
||||
39. CN model/cluster-file coupling: store cluster-file hash in the model; policy = warn (reference 1.1–1.3) vs error (reference 1.4) — make it
|
||||
configurable and test both; model must also match manifest revision (error otherwise). Model training requires ≥96 samples/90 passing; calling ≥24/22.
|
||||
40. Star-allele edge cases to test: unknown/undefined allele in phenotype lookup must not crash (CYP2E1 cluster-drift case); reference-allele
|
||||
genes (ABCG2) get default "Normal"; CYP2D6 exon-9 conversion (*36 vs *83+…) and hybrids/duplications (`*36x2+*83+*10`); CYP2A6 *1 definition sanity;
|
||||
HGVS REF>ALT formatting (never `_`); multi-allelic extension of bi-allelic DB definitions; many-probes-to-one-variant mapping; symlinked inputs;
|
||||
requirement that inputs be squashed + bgzipped + indexed (validate and error clearly).
|
||||
41. No-call handling: distinguish per-sample no-call variants from variants not on the array; list impacted alleles; for no-call diplotypes return
|
||||
ranked candidates with variant deltas and bound the edit distance; ensure JSON and CSV carry identical supporting evidence for **both** alleles.
|
||||
42. Scoring: probability-like score with population prior + raw score without; alternates retained if ≥ 50% of top; multiple tied top solutions
|
||||
joined by `;`; low-GS (<0.15) supporting-variant flag.
|
||||
43. Solver robustness: if an ILP/MIP formulation is used, guard against stale-solution/memory errors and make results solver-version independent.
|
||||
|
||||
### 5.8 QC module (§9)
|
||||
44. Control metrics and formulas match the reference table (bkg definitions, x = 3000, genotyping vs methylation variants); produce
|
||||
`controls_raw` / `controls_qc` equivalents; YAML thresholds with `null` = disabled; CLI overrides YAML overrides defaults (0.98 / 0.20);
|
||||
multi-dataset reports with labels; IDAT metadata (imaging date, plate, well, scanner ID/version) extracted; privacy warning when sample-sheet metadata flows into reports.
|
||||
45. HTML report scalability: >12,000 samples, >6 facet levels, long hover labels — test the plan's self-contained report at these sizes.
|
||||
|
||||
### 5.9 Methylation QC (plan "v2 methylation control QC")
|
||||
46. Reproduce noob + dye-bias correction + detection p-values (SeSAMe-equivalent), 21 control metrics with per-array thresholds, offset x configurable,
|
||||
PCA on betas; test sample counts around 80–100 with >900K-probe arrays (reference crash zone) and custom manifests lacking control sections (clear errors/NA metrics).
|
||||
|
||||
### 5.10 Release-management lessons
|
||||
47. Keep a machine-readable changelog of every default change (the reference changed CNV min size 0 → 20 → 25/50 kb and removed/renamed options
|
||||
across minor versions); emit all effective thresholds into output headers/JSON (the reference does this in cyto JSON) and into `run_manifest.json`.
|
||||
48. Keep output schemas versioned (the reference renamed JSON keys in 1.1 and 1.4, changed `annotationType` int→string, and shipped malformed keys
|
||||
such as `"chrom:"`); add JSON-schema validation tests to CI.
|
||||
Reference in new issue
Block a user