kemal/SimurgArray
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

P5a: QC — control metrics, thresholds, status, contamination, relatedness, locus summary

Browse Source 
- qc/controls: canonical control names (QC-002), bkg/offset (QC-003), 13 genotyping metrics (QC-004/005),
  missing/zero-divisor handling (QC-007/008); controls_raw.tsv / controls_qc.tsv (QC-009/011) incl. compat CSVs
- qc/thresholds + profiles: YAML schema, validation codes, precedence, shipped profiles, auto (QC-020..025)
- qc/status: functional PASS/FAIL/NA, control CLEAR/FLAGGED, flags (QC-030..033)
- qc/contamination: BAF regression; bit-compatible with gtc2vcf BAFregress (36/36, K-017), exact mode too
- qc/relatedness: KING-robust, duplicates, Mendelian errors, pedigree from sheet (IN-018)
- qc/locus_summary: OUT-072, bit-identical to BeadArrayFiles except ChiTest (K-018)
- qc/suggest: suggest-thresholds (QC-026)
- pipeline/qc: disk-backed cohort store; CLI qc controls|samples|contamination|relatedness|loci|suggest-thresholds
- genotype: control tables (OUT-070), het_rate, QC columns, cohort QC
- tests: unit + oracle (BAFregress plugin, BeadArrayFiles locus summary)
This commit is contained in:
kemal committed 2026-09-28 08:35:10 +01:00
1 parent 5d3e334ab7
commit eb06e9de04
28 files changed
+3355 -22
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

docs/known_unknowns.md
+3
View File
@@ -4,6 +4,9 @@ Plan §20 lists the open questions; this file records what the implementation ha
| ID | Date | Topic | Finding | Status |
|---|---|---|---|---|
| K-019 | 2026-09-28 | GSA demo relatedness | The 36-sample GSA v4 PGx demo set contains duplicate/replicate pairs (KING φ ≈ 0.50, concordance ≥ 0.9999) and first-degree relatives (φ ≈ 0.25); without a pedigree they are flagged `duplicate_of:`/`unexpected_relatedness` as designed. Useful as a small relatedness fixture. | documented |
| K-018 | 2026-09-28 | Locus-summary ChiTest_P100 | Against BeadArrayFiles `locus_summary.py` all columns are bit-identical on 36 GSA samples (683 054 loci) except ChiTest_P100, which differs by ≤ 3.6e-15 relative in ~15 % of loci: scipy 1.18's `ndtr` matches neither libm nor a cephes erf/erfc port bit-for-bit. Accepted (p-value). | accepted |
| K-017 | 2026-09-28 | BAF-regression median quirk | gtc2vcf `BAFregress` (1) compares AF with the threshold as a float (0.1f), and (2) for an even number of AA/BB calls averages the quickselect result with whatever element klib `ks_ksmall` leaves just below it, not the lower middle value. Both reproduced (`--median reference`, default): 36/36 GSA samples identical to the plugin at its printed precision; `--median exact` gives the true median. SimurgArray uses autosomal sites only in pipelines; the plugin uses every VCF record. | resolved |
| K-016 | 2026-09-28 | Sex rule of AutoCall 1.6.2.2 | Rule 1 (X heterozygosity) matches all 2141 vendor GTCs; rule 2 (X/Y intensity) differs in 12. | resolved |
| K-015 | 2026-09-28 | Legacy GTC v3 statistics | AutoCall 1.6.2.2 writes GC10 = GC50 = 0 and #calls = #no-calls = 0 in GTC v3 (it does fill LogR dev and #intensity-only). gtc2vcf's AutoConvert 1.6.3.1 preset fills them, so this is specific to 1.6.2.x. Emulated by `--preset autocall-1.6.2`; the sample summary still reports the real statistics. | resolved; cloud re-run pending |
| K-014 | 2026-09-28 | NaN sign | NaNs produced by libm (e.g. log2 of a negative extrapolated R) take the platform's default sign: + on aarch64, − on x86-64. SimurgArray always writes 0xFFC00000 (the reference's `-NAN`, x86 behaviour). The aarch64-built oracle differs in 1 locus of 9 GSA samples; oracle tests compare with NaNs canonicalised. | resolved |
docs/messages.md
+23
View File
@@ -25,6 +25,21 @@ Generated from `src/simurg/core/errors.py` — do not edit by hand.
| `E_NO_INPUT` | error | Neither an input folder nor a sample sheet was given | CLI-006 |
| `E_NO_INPUT_FOLDER` | error | Sample sheet has no RootFolder and no folder was given | CLI-006 |
| `E_OUTPUT_EXISTS` | error | Outputs already exist; use --force or --resume | CLI-010 |
| `E_QC_CONFIG_EXT` | error | QC config file must be .yaml or .yml | QC-021 |
| `E_QC_CONFIG_MISSING` | error | QC config file does not exist | QC-021 |
| `E_QC_CONFIG_PARSE` | error | QC config file is not valid YAML | QC-021 |
| `E_QC_CONFIG_STRUCTURE` | error | QC config lacks the qc/report (QC/Report) wrapper | QC-021 |
| `E_QC_DATASET_INCOMPLETE` | error | Dataset folder lacks a required file | QC-041 |
| `E_QC_FORMAT` | error | Invalid --output-format value | QC-063 |
| `E_QC_LABEL_COUNT` | error | Number of labels does not match number of datasets | QC-042 |
| `E_QC_LABEL_DUPLICATE` | error | Dataset labels are not unique | QC-042 |
| `E_QC_METHYLATION` | error | Methylation control metrics are not available yet (v2) | QC-006 |
| `E_QC_MIXED_TYPES` | error | Genotyping and methylation datasets cannot be mixed | QC-043 |
| `E_QC_NO_DATASETS` | error | No QC dataset folder found | QC-041 |
| `E_QC_PROFILE` | error | Unknown QC profile | QC-024 |
| `E_QC_THRESHOLD_KEY` | error | Unknown QC threshold key | QC-021 |
| `E_QC_THRESHOLD_RANGE` | error | QC threshold value out of range | QC-021 |
| `E_QC_THRESHOLD_VALUE` | error | QC threshold value is not numeric or null | QC-021 |
| `E_SHEET_BARCODE_FORMAT` | error | Invalid SentrixBarcode_A / SentrixPosition_A value | IN-007 |
| `E_SHEET_DELIMITER` | error | Sample sheet is not comma-delimited CSV | IN-001 |
| `E_SHEET_DUP_COLUMN` | error | Duplicate column (column names are case-insensitive) | IN-003 |
@@ -36,6 +51,9 @@ Generated from `src/simurg/core/errors.py` — do not edit by hand.
| `E_SHEET_RESERVED` | error | Sample sheet column collides with an output column | IN-005 |
| `E_SHEET_TOO_LONG` | error | Sample sheet name or value longer than 500 characters | IN-006 |
| `I_UNLISTED_IDATS` | info | IDAT pairs not listed in the sample sheet are ignored | IN-016 |
| `W_CTRL_DEFS_DIFFER` | warning | BPM and CSV control definitions differ; the BPM wins | QC-001 |
| `W_CTRL_MISSING` | warning | Controls needed by a metric are absent; metric not evaluable | QC-007 |
| `W_CTRL_ZERO_DIVISOR` | warning | Control metric has a zero denominator (NaN) | QC-008 |
| `W_DUP_IDAT_RESOLVED` | warning | Duplicate IDAT pair resolved by --on-duplicate-idat | IN-022 |
| `W_EGT_NAME` | warning | Cluster file name is not portable | IN-045 |
| `W_GTC_FOREIGN` | warning | GTC was produced by another tool | IN-031 |
@@ -45,6 +63,11 @@ Generated from `src/simurg/core/errors.py` — do not edit by hand.
| `W_IDAT_NO_SCAN_TIME` | warning | IDAT has no scan entry in its run info | IN-024 |
| `W_IDAT_UNPAIRED` | warning | IDAT channel file without its partner | IN-021 |
| `W_MANIFEST_NO_REFSTRAND` | warning | Manifest lacks RefStrand annotation | IN-040 |
| `W_QC_LABEL_SANITIZED` | warning | Dataset label was sanitized | QC-042 |
| `W_QC_NO_BAF` | warning | GTC has no B-allele frequencies; contamination not estimated | QC-064 |
| `W_QC_PEDIGREE` | warning | Parent1/Parent2/Replicate value does not name a sample in the set | IN-018 |
| `W_QC_TOO_FEW` | warning | Too few samples or sites for a cohort QC statistic | QC-026 |
| `W_QC_UNMATCHED_SAMPLES` | warning | Sample IDs present in one QC file but not the other | QC-040 |
| `W_SHEET_BARCODE_REPAIRED` | warning | Barcode written as a decimal was repaired | IN-007 |
| `W_SHEET_EMPTY_COLUMN` | warning | Columns with empty headers are ignored | IN-004 |
| `W_SHEET_ROOTFOLDER_OVERRIDDEN` | warning | CLI folder overrides the sheet RootFolder | IN-008 |
docs/validation/P5.md
+22
View File
@@ -0,0 +1,22 @@
# Phase P5 validation — QC
Status: **part 1 (local, GSA v4 PGx demo, 36 samples) passed**; part 2 (1000 Genomes Omni2.5, 2141 samples, cloud) pending.
## Part 1 — local, against open-source references (`tests/oracle/test_p5_qc.py`)
| Check | Reference | Result |
|---|---|---|
| BAF-regression contamination, 36 samples, all 683 054 loci as sites | gtc2vcf `bcftools +BAFregress` (MIT), built by `scripts/make_oracle_gtcs.sh` | 36/36 identical at the plugin's printed precision (`%.4f`), homozygous-site counts identical (K-017: float threshold 0.1f and klib quickselect median reproduced) |
| Locus summary, 683 054 loci | BeadArrayFiles `examples/locus_summary.py` (BSD-2), `scripts/make_oracle_qc.sh` | every column bit-identical except ChiTest_P100 (≤ 3.6e-15 relative, K-018) |
| Control metrics | QC-004 formulas | unit tests with hand-computed values; GSA demo values plausible (staining, extension, NP > 5; hyb > 1) |
| Thresholds / profiles / status / flags | QC-020 … QC-033 | unit tests for every QC-021 error code, precedence, aliases, profiles, both status axes |
| KING-robust kinship, Mendelian errors | Manichaikul 2010 | simulated trio: duplicate φ = 0.50, parent–child φ ≈ 0.25, unrelated ≈ 0, 0 Mendelian errors in the true trio |
| End to end | `simurg qc controls`, `simurg genotype` with QC | control tables identical between the two paths; QC columns filled |
GSA demo observations: all 36 samples PASS (autosomal call rate ≥ 0.997, LogR dev 0.13–0.17), control status CLEAR;
contamination 0.0012–0.0040; replicate pairs (φ ≈ 0.50) and first-degree relatives (φ ≈ 0.25) are present (K-019).
## Part 2 — cohort scale (pending)
1000 Genomes Omni2.5, 2141 samples: relatedness and Mendelian errors against the 1000G pedigree (trios), sex against the
pedigree, contamination and control-metric distributions, runtime/memory of the disk-backed cohort store.