PGx CN: pgx copy-number train|call (CLI-031/032, PCN-001..017, IN-050), region BED v1, demo identities (K-028)

- simurg/pgx/cn.py: per-region CN mixture, mu_c = delta + gamma*log2((c+1)/3), free lower CN-0 component,
  Hardy-Weinberg-constrained weights over haplotype copy numbers, EM from a grid of label starts.
- pipeline/pgx_cn.py: model file (.simurg-pgxcn), batch rules (PCN-004), per-sample CN VCF + tbi, batch and
  sample summaries.
- The GSA v4 PGx demo arrays are 30 CEPH/1000G individuals (scripts/pgx/demo36_identity.tsv); against NYGC 30x
  SV RD_CN 224/240 region calls agree (GSTM1, UGT2B17 30/30), in-sample (docs/validation/V2_pgx_cn.md).
This commit is contained in:
kemal committed 2026-09-29 13:16:19 +01:00
1 parent 8c7414e894
commit cc8c934167
14 files changed
+1306 -1

No files matched your search

+1 -1
View File
@@ -13,7 +13,7 @@ used only as external subprocess oracles and are not copied or linked.
| [Illumina GTCtoVCF](https://github.com/Illumina/GTCtoVCF) — Copyright Illumina, Inc. | Apache-2.0 | Indel source-sequence handling ported to `simurg/harmonize/sites.py` (design-strand orientation, left shift, IUPAC flank matching, deletion/insertion context score); multi-probe consistency idea in `simurg/harmonize/combine.py`; test oracle (run under Python 3 with two local compatibility patches, never shipped) |
| [Picard `picard.arrays`](https://github.com/broadinstitute/picard) | MIT | Second format reference |
| [PyO3](https://github.com/PyO3/pyo3), [rust-numpy](https://github.com/PyO3/rust-numpy), [ndarray](https://github.com/rust-ndarray/ndarray) and their Rust dependencies | MIT / Apache-2.0 / BSD-2-Clause | Compiled into the optional `simurg_core` extension (`rust/simurg-core`); list with `cargo tree` |
| [UCSC Genome Browser](https://genome.ucsc.edu/) `cytoBand` tables (hg38, hg19) and `ncbiRefSeq` (RefSeq transcripts, NCBI) | Free for all uses (UCSC data terms; RefSeq is public domain) | `simurg/cn/data/cytoBand_GRCh3{7,8}.txt.gz` and the derived centromere/acrocentric mask BEDs are shipped; `scripts/build_annotation_bundle.sh` downloads cytoBand + ncbiRefSeq for gene lists |
| [UCSC Genome Browser](https://genome.ucsc.edu/) `cytoBand` tables (hg38, hg19) and `ncbiRefSeq` (RefSeq transcripts, NCBI) | Free for all uses (UCSC data terms; RefSeq is public domain) | `simurg/cn/data/cytoBand_GRCh3{7,8}.txt.gz` and the derived centromere/acrocentric mask BEDs are shipped; `scripts/build_annotation_bundle.sh` downloads cytoBand + ncbiRefSeq for gene lists; `simurg/pgx/data/pgx_cn_regions.GRCh38.v1.bed` is derived from ncbiRefSeq exon coordinates (`scripts/pgx/build_cn_regions.py`) |
| [illuminaio](https://github.com/HenrikBengtsson/illuminaio) | GPL-2 | Documentation of IDAT field codes only; oracle via subprocess |
## BSD-2-Clause (BeadArrayFiles, verbatim from its README)
+1
View File
@@ -4,6 +4,7 @@ Plan §20 lists the open questions; this file records what the implementation ha
| ID | Date | Topic | Finding | Status |
|---|---|---|---|---|
| K-028 | 2026-09-29 | PGx CN (v2) | Deviations from PCN-009: the per-probe reference is the training cohort's median stored in the model (a calling-batch median is not CN 2 where deletions are common, e.g. GSTM1 null ≈ 50 %); no GC-correction step; the CN mixture is ours: µ_c = δ + γ·log2((c+1)/3) per region with a free lower CN-0 component and Hardy–Weinberg-constrained weights (probe compression differs by region: γ ≈ 0.19 CYP2D6.p5 … 0.56 GSTT1). Interim validation on the GSA v4 PGx demo — whose 36 arrays turned out to be 30 CEPH/1000G individuals — against NYGC 30x SV RD_CN: 224/240 region calls (GSTM1, UGT2B17 30/30), but in-sample (trained on the same arrays, < 96 samples, priors chosen with this truth in view). | Open: validate on an independent GSA-PGx cohort; CYP2D6 via GeT-RM star alleles (`docs/validation/V2_pgx_cn.md`). |
| K-027 | 2026-09-29 | CN/LOH (v2) | CYTO-014 noise gate: the summary `logr_dev` also measures cluster-file mismatch (GSE165845: 0.246, all samples `SampleQuality`) while the LRR used after the batch reference has SD ≈ 0.1; the gate now uses the SD of the normalised LRR (the JSON still reports the summary value). Spec: built-in mask/cytobands from UCSC; the annotation bundle uses UCSC ncbiRefSeq instead of RefSeq GFF + Ensembl GTF + HGNC. Real-data result: constitutional DEL/DUP calls 74–77 % in NYGC truth, mosaic calls mostly false positives. Pi tuning: mosaic prior / switch distance barely matter; CYTO-041 (noise-normalised BAF split, complete-loss promotion) cuts PASS mosaic calls by ~30 % without losing implanted ≥ 15 % mosaics. Two limits remain: (1) without phasing, < 20 % mosaic losses are not separable from regional BAF noise; (2) common CNVs (carrier frequency > 20 %) are almost never detected (1/236 DEL) because the median-LRR batch reference absorbs them; rare DEL (< 5 %) 38 %. | Open: rare-event sensitivity as primary metric; phasing-based mosaic BAF (`docs/validation/V2_cn.md`). |
| K-026 | 2026-09-29 | Methylation QC (v2) | Spec deviations: vendor EPIC manifests put a 5th field on some control lines (≥ 4 fields accepted); detection p follows sesame's pOOBAH (1 − max(F(M), F(U))) rather than the ambiguous "1 − ECDF(M + U)"; minfi-style `Sentrix_ID`/`Sentrix_Position` sheet keys accepted. Against sesame 1.28.1 on minfiDataEPIC: β r = 0.99998 (max \|Δβ\| 0.011), detection calls 99.1 % identical, noob signals within 0.5 % (median) / 5 % (max). The systematic remainder is unexplained; it is not the negative controls (sesame did not attach the supplied control table). | Open; `docs/validation/V2_methylation.md`. Next step: dump sesame's background vectors and noob parameters to find the difference. |
| K-025 | 2026-09-28 | Scaling limits (P9) | With the Rust kernels a Graviton3 core genotypes 78–80 GSA samples/min. At > 8 workers a 125 MB/s gp3 volume caps the run at ~620 samples/min (GTC ≈ 13 MB each); on a RAM disk efficiency is 0.77 at 16 and 0.48 at 32 workers for 360 samples, explained by a fixed start-up cost (≈ 2.5 s + 0.2 s per worker). Linear scaling for large batches is inferred from that model, not measured. | `docs/validation/P9.md`; measure a ≥ 5000-sample batch when one is at hand. |
+9
View File
@@ -36,11 +36,15 @@ Generated from `src/simurg/core/errors.py` — do not edit by hand.
| `E_METH_NO_CONTROLS` | error | Missing control probes in manifest | MET-002 |
| `E_METH_NO_IDATS` | error | No IDATs found | MET-004 |
| `E_METH_PREP_FAILED` | error | Prep failed (sample) | MET-010 |
| `E_MODEL_CLUSTER_MISMATCH` | error | Input GTCs were called with another cluster file than the model | IN-050 |
| `E_MODEL_MANIFEST_MISMATCH` | error | Model was built for a different manifest probe set | IN-051 |
| `E_MODEL_VERSION` | error | Model schema is too old; rebuild it | IN-051 |
| `E_NO_INPUT` | error | Neither an input folder nor a sample sheet was given | CLI-006 |
| `E_NO_INPUT_FOLDER` | error | Sample sheet has no RootFolder and no folder was given | CLI-006 |
| `E_OUTPUT_EXISTS` | error | Outputs already exist; use --force or --resume | CLI-010 |
| `E_PGX_BATCH_TOO_SMALL` | error | Too few (passing) samples in the batch for PGx CN calling | PCN-004 |
| `E_PGX_CN_FAILED` | error | PGx CN calling failed for the sample | PCN-002 |
| `E_PGX_TRAIN_TOO_SMALL` | error | Too few (passing) samples to train a PGx CN model | PCN-005 |
| `E_QC_CONFIG_EXT` | error | QC config file must be .yaml or .yml | QC-021 |
| `E_QC_CONFIG_MISSING` | error | QC config file does not exist | QC-021 |
| `E_QC_CONFIG_PARSE` | error | QC config file is not valid YAML | QC-021 |
@@ -92,7 +96,12 @@ Generated from `src/simurg/core/errors.py` — do not edit by hand.
| `W_METH_MISSING_CONTROL_TYPES` | warning | Control types missing from the manifest; metric set to NA | MET-009 |
| `W_METH_MISSING_SAMPLE` | warning | Missing sample: IDATs absent; the sample fails, the run continues | MET-015 |
| `W_METH_NO_FEATURES` | warning | No features, skipping PCA plot | MET-013 |
| `W_MODEL_CLUSTER_MISMATCH` | warning | Input GTCs were called with another cluster file than the model | IN-050 |
| `W_NO_BGZIP` | warning | Uncompressed VCF written; downstream commands need bgzip + index | OUT-010 |
| `W_PGX_BATCH_BELOW_RECOMMENDED` | warning | Fewer passing samples than recommended | PCN-004 |
| `W_PGX_BATCH_COMPOSITION` | warning | Batch spans several lots/batches or > 14 days of scans | PCN-006 |
| `W_PGX_NO_ALT_CONTIG` | warning | FASTA lacks a contig carrying a PGx region; REF written as N | IN-062 |
| `W_PGX_REGION_FEW_PROBES` | warning | PGx CN region has fewer than 10 unmasked probes | PCN-008 |
| `W_QC_LABEL_SANITIZED` | warning | Dataset label was sanitized | QC-042 |
| `W_QC_NO_BAF` | warning | GTC has no B-allele frequencies; contamination not estimated | QC-064 |
| `W_QC_PEDIGREE` | warning | Parent1/Parent2/Replicate value does not name a sample in the set | IN-018 |
+48
View File
@@ -0,0 +1,48 @@
# V2 — Targeted PGx copy number (PCN-001 … PCN-017)
Status: **implemented, interim validation** (2026-09-29). `simurg pgx copy-number train|call` (CLI-031/032), model
file `.simurg-pgxcn` (IN-050), region BED v1 built from UCSC ncbiRefSeq (`scripts/pgx/build_cn_regions.py`,
packaged as `simurg/pgx/data/pgx_cn_regions.GRCh38.v1.bed`), per-sample `*.pgx_cn.vcf.gz` + `.tbi`,
`pgx_cn_batch_summary.tsv`, `pgx_cn_sample_summary.tsv`. Unit tests: `tests/unit/test_pgx_cn.py`.
## Data
* **Arrays:** the GSA v4 PGx demo (`idats_Demo_36`, 36 arrays), genotyped by SimurgArray. Its CNV probes cover every
region (51–357 probes; GSTT1: 160 probes on `chr22_KI270879v1_alt` in the GRCh38 manifest).
* **Identities:** the demo arrays are 30 CEPH / 1000 Genomes individuals (6 replicate arrays). Identified by
genotype concordance with the 3202-sample 1000G 30x panel on 4647 chr22 SNPs: best match 0.992–0.996, next best
≤ 0.91 (relatives) — `scripts/pgx/demo36_identity.tsv`.
* **Truth:** NYGC 30x SV callset (1KGP_3202 GATK-SV/svtools freeze V3), RD_CN per individual
(`scripts/pgx/validate_cn_truth.py`).
* The GSE165845 GSA-MD v1 arrays (360 arrays, CN/LOH validation) have 0–9 probes per PGx region: not usable here.
## Result (30 individuals, model trained on the same 36 arrays with `--min-samples 30`)
| Region | Concordance with NYGC RD_CN | Mismatches (truth → call) |
|---|---|---|
| GSTM1 (whole gene) | **30/30** | – |
| UGT2B17 (whole gene) | **30/30** | – |
| SULT1A1 | 29/30 | 2→1 |
| CYP2A6.exon.1 | 29/30 | 3→2 |
| CYP2A6.intron.7 | 28/30 | 3→2, 1→2 |
| CYP2D6.p5 | 28/30 | 4→3, 2→1 |
| CYP2D6.intron.2 | 25/30 | 3→2 ×3, 4→3, 2→1 |
| CYP2D6.exon.9 | 25/30 | 3→2 ×3, 4→2, 2→1 |
| **all** | **224/240 (93.3 %)** | |
GSTT1 (no SV truth on the alt contig): CN 0/1/2 = 5/16/9, in Hardy–Weinberg proportions (null allele 0.43; CEU
literature ≈ 0.4–0.5). Details: `V2_pgx_cn_demo36.json`.
* CYP2D6 truth is one gene-level record; the exon-9 / intron-2 "misses" at truth CN 3 are expected for tandem
structures such as `*68+*4` or `*36+*10`, where not every sub-region is duplicated. Star-allele truth (GeT-RM)
is the right yardstick and follows with the star-allele caller.
* PCN-017 target (≥ 95 % per region) is met for GSTM1, UGT2B17, SULT1A1, CYP2A6.exon.1; not (yet) shown for the
CYP2D6 sub-regions.
## Caveats (K-028)
* Trained and evaluated on the same 30 individuals (the spec requires ≥ 96 training samples; the priors were
chosen while looking at this truth). This is an optimistic, in-sample figure until an independent cohort is
available.
* Deviations from PCN-009: reference from the training cohort (not the calling batch); no GC-correction step; CN
model = compressed-intensity template + HWE-constrained weights (`simurg/pgx/cn.py`).
+89
View File
@@ -0,0 +1,89 @@
{
"individuals": 30,
"regions": {
"GSTM1": {
"record": "HGSV_9691",
"concordant": 30,
"n": 30,
"rate": 1.0,
"truth_to_call_mismatches": {}
},
"UGT2B17": {
"record": "HGSV_63272",
"concordant": 30,
"n": 30,
"rate": 1.0,
"truth_to_call_mismatches": {}
},
"SULT1A1": {
"record": "HGSV_207805",
"concordant": 29,
"n": 30,
"rate": 0.9667,
"truth_to_call_mismatches": {
"2->1": 1
}
},
"CYP2A6.intron.7": {
"record": "HGSV_232737",
"concordant": 28,
"n": 30,
"rate": 0.9333,
"truth_to_call_mismatches": {
"3->2": 1,
"1->2": 1
}
},
"CYP2A6.exon.1": {
"record": "HGSV_232737",
"concordant": 29,
"n": 30,
"rate": 0.9667,
"truth_to_call_mismatches": {
"3->2": 1
}
},
"CYP2D6.exon.9": {
"record": "HGSV_247317",
"concordant": 25,
"n": 30,
"rate": 0.8333,
"truth_to_call_mismatches": {
"4->2": 1,
"3->2": 3,
"2->1": 1
}
},
"CYP2D6.intron.2": {
"record": "HGSV_247317",
"concordant": 25,
"n": 30,
"rate": 0.8333,
"truth_to_call_mismatches": {
"4->3": 1,
"3->2": 3,
"2->1": 1
}
},
"CYP2D6.p5": {
"record": "HGSV_247317",
"concordant": 28,
"n": 30,
"rate": 0.9333,
"truth_to_call_mismatches": {
"4->3": 1,
"2->1": 1
}
}
},
"overall": {
"concordant": 224,
"n": 240,
"rate": 0.9333
},
"gstt1_cn_distribution": {
"0": 5,
"1": 16,
"2": 9
}
}
+85
View File
@@ -0,0 +1,85 @@
#!/usr/bin/env python3
"""Build the PGx CN target-region BED (spec PCN-001) from UCSC ncbiRefSeq (GRCh38).
python scripts/pgx/build_cn_regions.py ncbiRefSeq.hg38.txt.gz OUT.bed [--version N]
Regions (0-based half-open BED: chrom start0 end region gene), from the MANE/first-listed transcript:
CYP2D6.p5 2 kb 5' flank of CYP2D6 (upstream of exon 1; the gene is on the minus strand)
CYP2D6.intron.2 CYP2D6 intron 2
CYP2D6.exon.9 CYP2D6 exon 9
CYP2A6.exon.1 CYP2A6 exon 1 (± 500 bp)
CYP2A6.intron.7 CYP2A6 intron 7
GSTM1, GSTT1 (on chr22_KI270879v1_alt, IN-062), UGT2B17, SULT1A1, CYP2E1: whole gene.
Exons and introns are numbered in transcript order (strand-aware). [DB: NCBI RefSeq via UCSC ncbiRefSeq]
"""
from __future__ import annotations
import argparse
import gzip
from pathlib import Path
TRANSCRIPT = {"CYP2D6": "NM_000106", "CYP2A6": "NM_000762", "GSTM1": "NM_000561", "GSTT1": "NM_000853",
"UGT2B17": "NM_001077", "SULT1A1": "NM_001055", "CYP2E1": "NM_000773"} # fmt: skip
FLANK = 2000
PAD = 500
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("refseq", type=Path)
ap.add_argument("out", type=Path)
ap.add_argument("--version", type=int, default=1)
a = ap.parse_args()
tx: dict[str, tuple[str, str, list[int], list[int]]] = {}
with gzip.open(a.refseq, "rt") as fh:
for line in fh:
f = line.rstrip("\n").split("\t")
name, chrom, strand, gene = f[1], f[2], f[3], f[12]
if gene not in TRANSCRIPT or not name.startswith(TRANSCRIPT[gene] + "."):
continue
if "_" in chrom and not (gene == "GSTT1" and chrom == "chr22_KI270879v1_alt"):
continue # primary assembly, except GSTT1 which is only on the alt contig
starts = [int(x) for x in f[9].rstrip(",").split(",")]
ends = [int(x) for x in f[10].rstrip(",").split(",")]
tx.setdefault(gene, (chrom, strand, starts, ends))
missing = set(TRANSCRIPT) - set(tx)
if missing:
raise SystemExit(f"transcripts not found: {sorted(missing)}")
rows: list[tuple[str, int, int, str, str]] = []
def exon(gene: str, k: int) -> tuple[int, int]:
chrom, strand, s, e = tx[gene]
i = k - 1 if strand == "+" else len(s) - k
return s[i], e[i]
def intron(gene: str, k: int) -> tuple[int, int]:
chrom, strand, s, e = tx[gene]
a_, b_ = exon(gene, k), exon(gene, k + 1)
return (a_[1], b_[0]) if strand == "+" else (b_[1], a_[0])
for gene in ("CYP2D6", "CYP2A6"):
chrom, strand, s, e = tx[gene]
if gene == "CYP2D6":
p5 = (e[-1], e[-1] + FLANK) if strand == "-" else (s[0] - FLANK, s[0])
rows += [(chrom, *p5, "CYP2D6.p5", gene), (chrom, *intron(gene, 2), "CYP2D6.intron.2", gene),
(chrom, *exon(gene, 9), "CYP2D6.exon.9", gene)] # fmt: skip
else:
x1 = exon(gene, 1)
rows += [(chrom, x1[0] - PAD, x1[1] + PAD, "CYP2A6.exon.1", gene),
(chrom, *intron(gene, 7), "CYP2A6.intron.7", gene)] # fmt: skip
for gene in ("GSTM1", "GSTT1", "UGT2B17", "SULT1A1", "CYP2E1"):
chrom, strand, s, e = tx[gene]
rows.append((chrom, s[0], e[-1], gene, gene))
order = {f"chr{i}": i for i in range(1, 23)}
rows.sort(key=lambda r: (order.get(r[0].split("_")[0], 99), r[0], r[1]))
with a.out.open("w") as fh:
fh.write(f"# simurg PGx CN regions v{a.version}, GRCh38, from UCSC ncbiRefSeq (PCN-001)\n")
for r in rows:
fh.write("\t".join(map(str, r)) + "\n")
print(a.out.read_text())
return 0
if __name__ == "__main__":
raise SystemExit(main())
+39
View File
@@ -0,0 +1,39 @@
# GSA v4 PGx demo (idats_Demo_36) sample identities: best match among the 3202 1000 Genomes 30x individuals
# by genotype concordance on 4647 chr22 SNPs with 1000G AF 0.05-0.95 (next best <= 0.91: relatives)
sample_id individual concordance second_best second_concordance
207895980019_R03C01 NA12875 0.9920 NA12865 0.7038
207895980019_R03C02 NA12156 0.9933 NA10831 0.7021
207895980019_R04C01 NA12156 0.9931 NA10831 0.7027
207895980019_R05C01 NA12044 0.9931 NA10857 0.6969
207895980019_R06C01 NA11992 0.9931 NA10860 0.6848
207895980019_R07C01 NA06993 0.9940 NA06991 0.7209
207895980019_R07C03 NA07019 0.9953 NA07056 0.7051
207895980019_R08C01 NA12239 0.9938 NA10847 0.7022
207895980019_R09C01 NA12716 0.9944 NA12707 0.6896
207895980019_R09C03 NA06991 0.9931 NA06993 0.7215
207895980019_R10C01 NA12878 0.9933 NA12892 0.7022
207895980019_R11C01 NA12760 0.9948 NA12752 0.7171
207895980019_R11C02 NA06985 0.9948 NA06991 0.6996
207895980019_R12C01 NA11993 0.9961 NA10860 0.7168
207895980019_R12C02 NA12003 0.9935 NA10838 0.6998
207895980019_R13C03 NA06994 0.9948 NA07029 0.7114
207895980019_R14C03 NA11993 0.9959 NA10860 0.7163
207895980019_R15C02 NA12003 0.9931 NA10838 0.6993
207895980035_R03C03 NA07029 0.9952 NA07000 0.7237
207895980035_R04C03 NA12892 0.9957 NA12878 0.7005
207895980035_R05C03 NA12248 0.9950 NA10835 0.7143
207895980035_R06C03 NA10846 0.9920 NA12145 0.7056
207895980035_R07C01 NA07000 0.9957 NA07029 0.7231
207895980035_R07C03 NA12155 0.9935 NA10831 0.6903
207895980035_R08C01 NA11832 0.9942 NA10855 0.7252
207895980035_R08C03 NA06993 0.9940 NA06991 0.7205
207895980035_R09C03 NA10838 0.9948 NA12004 0.7007
207895980035_R10C01 NA12813 0.9937 NA07045 0.9103
207895980035_R10C03 NA12249 0.9938 NA10835 0.6858
207895980035_R11C03 NA12812 0.9942 NA12801 0.7061
207895980035_R12C03 NA11881 0.9935 NA10859 0.7138
207895980035_R13C01 NA11995 0.9953 NA10861 0.6857
207895980035_R13C03 NA12248 0.9950 NA10835 0.7145
207895980035_R14C01 NA12891 0.9942 NA12878 0.6998
207895980035_R15C03 NA12043 0.9946 NA10857 0.7162
207895980042_R15C02 NA11832 0.9942 NA10855 0.7253
1 # GSA v4 PGx demo (idats_Demo_36) sample identities: best match among the 3202 1000 Genomes 30x individuals
2 # by genotype concordance on 4647 chr22 SNPs with 1000G AF 0.05-0.95 (next best <= 0.91: relatives)
3 sample_id individual concordance second_best second_concordance
4 207895980019_R03C01 NA12875 0.9920 NA12865 0.7038
5 207895980019_R03C02 NA12156 0.9933 NA10831 0.7021
6 207895980019_R04C01 NA12156 0.9931 NA10831 0.7027
7 207895980019_R05C01 NA12044 0.9931 NA10857 0.6969
8 207895980019_R06C01 NA11992 0.9931 NA10860 0.6848
9 207895980019_R07C01 NA06993 0.9940 NA06991 0.7209
10 207895980019_R07C03 NA07019 0.9953 NA07056 0.7051
11 207895980019_R08C01 NA12239 0.9938 NA10847 0.7022
12 207895980019_R09C01 NA12716 0.9944 NA12707 0.6896
13 207895980019_R09C03 NA06991 0.9931 NA06993 0.7215
14 207895980019_R10C01 NA12878 0.9933 NA12892 0.7022
15 207895980019_R11C01 NA12760 0.9948 NA12752 0.7171
16 207895980019_R11C02 NA06985 0.9948 NA06991 0.6996
17 207895980019_R12C01 NA11993 0.9961 NA10860 0.7168
18 207895980019_R12C02 NA12003 0.9935 NA10838 0.6998
19 207895980019_R13C03 NA06994 0.9948 NA07029 0.7114
20 207895980019_R14C03 NA11993 0.9959 NA10860 0.7163
21 207895980019_R15C02 NA12003 0.9931 NA10838 0.6993
22 207895980035_R03C03 NA07029 0.9952 NA07000 0.7237
23 207895980035_R04C03 NA12892 0.9957 NA12878 0.7005
24 207895980035_R05C03 NA12248 0.9950 NA10835 0.7143
25 207895980035_R06C03 NA10846 0.9920 NA12145 0.7056
26 207895980035_R07C01 NA07000 0.9957 NA07029 0.7231
27 207895980035_R07C03 NA12155 0.9935 NA10831 0.6903
28 207895980035_R08C01 NA11832 0.9942 NA10855 0.7252
29 207895980035_R08C03 NA06993 0.9940 NA06991 0.7205
30 207895980035_R09C03 NA10838 0.9948 NA12004 0.7007
31 207895980035_R10C01 NA12813 0.9937 NA07045 0.9103
32 207895980035_R10C03 NA12249 0.9938 NA10835 0.6858
33 207895980035_R11C03 NA12812 0.9942 NA12801 0.7061
34 207895980035_R12C03 NA11881 0.9935 NA10859 0.7138
35 207895980035_R13C01 NA11995 0.9953 NA10861 0.6857
36 207895980035_R13C03 NA12248 0.9950 NA10835 0.7145
37 207895980035_R14C01 NA12891 0.9942 NA12878 0.6998
38 207895980035_R15C03 NA12043 0.9946 NA10857 0.7162
39 207895980042_R15C02 NA11832 0.9942 NA10855 0.7253
+72
View File
@@ -0,0 +1,72 @@
#!/usr/bin/env python3
"""Validate `simurg pgx copy-number call` against the NYGC 30x SV callset (1KGP_3202 freeze V3) — PCN-017.
python scripts/pgx/validate_cn_truth.py --calls CALL_DIR --identity scripts/pgx/demo36_identity.tsv \
--sv 1KGP_3202.sv_freeze_V3.vcf.gz --bcftools BCFTOOLS --out summary.json
Truth per region = RD_CN of one SV record per region (below, chosen as the record spanning the whole region; for
GSTM1 the read-depth DUP/CNV record, whose RD_CN gives 0/1/2 in Hardy–Weinberg proportions — the DEL record there
reports CN 1 for 27/30 individuals). CYP2D6 sub-regions share one gene-level record, so exon-9 conversions (*36) and
2D7-2D6 hybrids (*13, *68) legitimately differ from it. GSTT1 lies on an alt contig with no SV calls: only its CN
distribution is reported. Replicate arrays of one individual are counted once (the first array).
"""
from __future__ import annotations
import argparse
import collections
import csv
import json
import subprocess
from pathlib import Path
TRUTH_RECORD = {"GSTM1": "HGSV_9691", "UGT2B17": "HGSV_63272", "SULT1A1": "HGSV_207805",
"CYP2A6.intron.7": "HGSV_232737", "CYP2A6.exon.1": "HGSV_232737", "CYP2D6.exon.9": "HGSV_247317",
"CYP2D6.intron.2": "HGSV_247317", "CYP2D6.p5": "HGSV_247317"} # fmt: skip
def main() -> int:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--calls", required=True, type=Path)
ap.add_argument("--identity", required=True, type=Path)
ap.add_argument("--sv", required=True)
ap.add_argument("--bcftools", default="bcftools")
ap.add_argument("--out", required=True, type=Path)
a = ap.parse_args()
ident = {}
with a.identity.open() as fh:
for r in csv.DictReader((ln for ln in fh if not ln.startswith("#")), delimiter="\t"):
ident[r["sample_id"]] = r["individual"]
calls: dict[str, dict[str, str]] = {}
with (a.calls / "pgx_cn_sample_summary.tsv").open() as fh:
for r in csv.DictReader(fh, delimiter="\t"):
ind = ident.get(r["sample_id"])
if ind and ind not in calls:
calls[ind] = {k[3:]: v for k, v in r.items() if k.startswith("cn_")}
ids = sorted(set(TRUTH_RECORD.values()))
q = subprocess.run([a.bcftools, "query", "-H", "-s", ",".join(sorted(calls)), "--force-samples", "-i",
"ID=@/dev/stdin", "-f", "%ID[\t%RD_CN]\n", a.sv], input="\n".join(ids), capture_output=True,
text=True, check=True).stdout.splitlines() # fmt: skip
samples = [h.split("]")[1].split(":")[0] for h in q[0].split("\t")[1:]]
truth = {ln.split("\t")[0]: dict(zip(samples, ln.split("\t")[1:], strict=True)) for ln in q[1:]}
res: dict[str, dict[str, object]] = {}
tot = [0, 0]
for region, rid in TRUTH_RECORD.items():
pairs = [(truth[rid][ind], c[region]) for ind, c in calls.items() if ind in truth.get(rid, {})]
pairs = [(t, c) for t, c in pairs if t not in (".", "")]
ok = sum(t == c for t, c in pairs)
tot[0] += ok
tot[1] += len(pairs)
res[region] = {"record": rid, "concordant": ok, "n": len(pairs),
"rate": round(ok / len(pairs), 4) if pairs else None,
"truth_to_call_mismatches": dict(collections.Counter(f"{t}->{c}" for t, c in pairs if t != c))} # fmt: skip
summary = {"individuals": len(calls), "regions": res, "overall": {"concordant": tot[0], "n": tot[1],
"rate": round(tot[0] / tot[1], 4) if tot[1] else None},
"gstt1_cn_distribution": dict(sorted(collections.Counter(c.get("GSTT1", ".") for c in calls.values()).items()))} # fmt: skip
a.out.write_text(json.dumps(summary, indent=1))
print(json.dumps(summary, indent=1))
return 0
if __name__ == "__main__":
raise SystemExit(main())
+72
View File
@@ -803,6 +803,78 @@ def cn_annotate_cmd(
run_guarded(rep, lambda: run_cn_annotate(opts, rep))
# --------------------------------------------------------------------------------------------------------
pgx_app = typer.Typer(help="Pharmacogenomics: targeted copy number, star alleles, phenotypes (v2).")
app.add_typer(pgx_app, name="pgx")
pgx_cn_app = typer.Typer(help="Targeted PGx copy number (PCN-001).")
pgx_app.add_typer(pgx_cn_app, name="copy-number")
@pgx_cn_app.command("train")
def pgx_cn_train_cmd(
gtcs: Annotated[Path, typer.Option("--gtcs", help="Folder searched recursively for training GTCs (required).")],
bpm: Annotated[Path, typer.Option("--bpm", help="BPM manifest (required).")],
fasta: Annotated[Path, typer.Option("--fasta", help="GRCh38 FASTA (required).")],
out: Annotated[Path, typer.Option("--out", help="Output folder.")] = Path("."),
mask: Annotated[Path | None, typer.Option("--mask", help="Probe mask TSV (IN-052).")] = None,
regions: Annotated[Path | None, typer.Option("--regions", help="Region BED (default: built-in v1).")] = None,
summary: Annotated[Path | None, typer.Option("--summary", help="sample_summary.tsv (assay control).")] = None,
platform: Annotated[str, typer.Option("--platform", help="auto|LCG|EX")] = "auto",
min_samples: Annotated[int, typer.Option("--min-samples", min=1)] = 96,
min_passing: Annotated[int, typer.Option("--min-passing", min=1)] = 90,
max_samples: Annotated[int, typer.Option("--max-samples", min=1)] = 3000,
train_lrr_sd_max: Annotated[float, typer.Option("--train-lrr-sd-max")] = 0.20,
train_call_rate_min: Annotated[float, typer.Option("--train-call-rate-min")] = 0.99,
train_assay_control_min: Annotated[float, typer.Option("--train-assay-control-min")] = 1.0,
force: ForceOpt = False,
log_json: LogJson = False,
debug: Debug = False,
quiet: Quiet = False,
) -> None:
"""Train a PGx CN model (CLI-032, PCN-005)."""
from simurg.pipeline.pgx_cn import PgxTrainOptions, run_train
if platform not in ("auto", "LCG", "EX"):
raise typer.BadParameter("--platform auto|LCG|EX")
rep = _reporter("pgx copy-number train", out, log_json, debug, quiet)
o = PgxTrainOptions(gtcs=gtcs, bpm=bpm, fasta=fasta, out=out, mask=mask, regions=regions, summary=summary,
platform=platform, min_samples=min_samples, min_passing=min_passing, max_samples=max_samples,
lrr_sd_max=train_lrr_sd_max, call_rate_min=train_call_rate_min,
assay_control_min=train_assay_control_min, force=force) # fmt: skip
run_guarded(rep, lambda: run_train(o, rep))
@pgx_cn_app.command("call")
def pgx_cn_call_cmd(
gtcs: Annotated[Path, typer.Option("--gtcs", help="Folder searched recursively for GTC files (required).")],
model: Annotated[Path, typer.Option("--model", help="PGx CN model (.simurg-pgxcn, required).")],
fasta: Annotated[Path, typer.Option("--fasta", help="GRCh38 FASTA with alt contigs (required).")],
out: Annotated[Path, typer.Option("--out", help="Output folder (required).")],
min_batch: Annotated[int, typer.Option("--min-batch", min=1)] = 24,
min_passing: Annotated[int, typer.Option("--min-passing", min=1)] = 22,
recommended_passing: Annotated[int, typer.Option("--recommended-passing", min=1)] = 96,
lrr_sd_max: Annotated[float, typer.Option("--lrr-sd-max")] = 0.20,
cluster_mismatch: Annotated[str, typer.Option("--cluster-mismatch", help="error|warn")] = "error",
no_bgzip: Annotated[bool, typer.Option("--no-bgzip")] = False,
force: ForceOpt = False,
log_json: LogJson = False,
debug: Debug = False,
quiet: Quiet = False,
) -> None:
"""Call targeted PGx copy number per region (CLI-031, PCN-009)."""
from simurg.pipeline.pgx_cn import PgxCallOptions, run_call
if cluster_mismatch not in ("error", "warn"):
raise typer.BadParameter("--cluster-mismatch error|warn")
if min_passing > min_batch:
raise typer.BadParameter("--min-passing must be <= --min-batch")
rep = _reporter("pgx copy-number call", out, log_json, debug, quiet)
o = PgxCallOptions(gtcs=gtcs, model=model, fasta=fasta, out=out, min_batch=min_batch, min_passing=min_passing,
recommended_passing=recommended_passing, lrr_sd_max=lrr_sd_max,
cluster_mismatch=cluster_mismatch, bgzip=not no_bgzip, force=force) # fmt: skip
run_guarded(rep, lambda: run_call(o, rep))
# --------------------------------------------------------------------------------------------------------
methyl_app = typer.Typer(help="Methylation arrays: control QC, detection, beta values (MET-001).")
app.add_typer(methyl_app, name="methyl")
+20
View File
@@ -151,6 +151,26 @@ CATALOGUE: dict[str, MessageDef] = dict(
_m("E_CN_SIDECAR_MISSING", "error", "cn_call_manifest.json not found in the cn call output", "CYTO-032"),
_m("E_CN_VCF_UNINDEXED", "error", "CNV VCFs must be bgzipped and indexed for annotate", "CYTO-032"),
_m("E_CN_FAILED", "error", "CN calling failed for the sample", "CYTO-040"),
# --- targeted PGx copy number (PCN-001 .. PCN-017) ---------------------------------------------------
_m("E_PGX_BATCH_TOO_SMALL", "error", "Too few (passing) samples in the batch for PGx CN calling", "PCN-004"),
_m("W_PGX_BATCH_BELOW_RECOMMENDED", "warning", "Fewer passing samples than recommended", "PCN-004"),
_m("E_PGX_TRAIN_TOO_SMALL", "error", "Too few (passing) samples to train a PGx CN model", "PCN-005"),
_m(
"E_MODEL_CLUSTER_MISMATCH",
"error",
"Input GTCs were called with another cluster file than the model",
"IN-050",
),
_m(
"W_MODEL_CLUSTER_MISMATCH",
"warning",
"Input GTCs were called with another cluster file than the model",
"IN-050",
),
_m("W_PGX_REGION_FEW_PROBES", "warning", "PGx CN region has fewer than 10 unmasked probes", "PCN-008"),
_m("W_PGX_NO_ALT_CONTIG", "warning", "FASTA lacks a contig carrying a PGx region; REF written as N", "IN-062"),
_m("W_PGX_BATCH_COMPOSITION", "warning", "Batch spans several lots/batches or > 14 days of scans", "PCN-006"),
_m("E_PGX_CN_FAILED", "error", "PGx CN calling failed for the sample", "PCN-002"),
# --- methylation control QC (MET-001 .. MET-018) -------------------------------------------------
_m("E_METH_NO_ASSAY", "error", "Probe section not found", "MET-002"),
_m("E_METH_NO_CONTROLS", "error", "Missing control probes in manifest", "MET-002"),
+204
View File
@@ -0,0 +1,204 @@
"""Targeted PGx copy number — spec PCN-001, PCN-009. Pure functions, no I/O.
Region value x = median over the region's unmasked probes of (sample-centred LRR − the probe's training reference),
minus the sample's scale (median of the same quantity over non-target autosomal probes).
CN mixture [OURS; array intensity compression as in cnvPartition/PennCNV, LIT: Wang et al. 2007 Genome Res 17:1665]:
component means µ_c = δ_r + γ_r · log2((c + 1) / 3) for CN c = 0 … 5 — a probe signal proportional to (c + β) with
a background β = 1 copy-equivalent, compressed by γ_r. δ_r (the region's CN-2 location), γ_r, σ_r and the weights
π_r are per region; γ_r has a Gaussian prior N(0.25, 0.08²). CN 0 may sit lower than the template (its own shift
ε₀ ≤ 0 and SD): GSTT1 probes drop by ~2 at CN 0, GSTM1 probes by ~0.3. The weights are constrained to Hardy–Weinberg
proportions of haplotype copy numbers (0 … 3 copies per haplotype, Dirichlet prior favouring one copy). Fitted by
EM from a grid of starts (each observed cluster labelled CN 0 … 4, CN 0 on the template or on the lowest values).
HWE and the template's spacing ratios identify the labels in multi-cluster regions (GSTM1 0/1/2 rather than 1/2/3,
which would imply unobserved CN-0 samples). On GSA v4 PGx CNV probes γ_r ≈ 0.19–0.56 (K-028).
"""
from __future__ import annotations
import math
from dataclasses import dataclass
import numpy as np
import numpy.typing as npt
F64 = npt.NDArray[np.float64]
CN_MAX = 5 # "5 or more"
CNS = np.arange(CN_MAX + 1)
TEMPLATE = np.log2((CNS + 1) / 3.0)
HAP_ALPHA = np.array([1.2, 3.0, 1.2, 1.05]) # Dirichlet prior on haplotype copy frequencies f_0..f_3 (f_1 favoured)
SIGMA_FLOOR = 0.012
GAMMA_PRIOR = (0.25, 0.08) # mean, SD
GAMMA_RANGE = (0.15, 0.80)
PI_FLOOR = 0.002
CN0_PRIOR_N = 2.0
@dataclass(frozen=True)
class Region:
chrom: str
start0: int
end: int
name: str
gene: str
def parse_regions(text: str) -> list[Region]:
"""PCN-001 BED: ``chrom start0 end region gene``; ``#`` comments."""
out = []
for ln in text.splitlines():
if not ln.strip() or ln.startswith("#"):
continue
c, s, e, r, g = ln.split("\t")[:5]
out.append(Region(c, int(s), int(e), r, g))
return out
@dataclass
class RegionFit:
delta: float
gamma: float
sigma: float
weights: list[float]
n: int
cn0_shift: float = 0.0 # ≤ 0: extra drop of the CN-0 component below the template (probe-specific background)
cn0_sigma: float = 0.0 # SD of the CN-0 component (≥ sigma; 0 = sigma)
def means(self) -> F64:
m = self.delta + self.gamma * TEMPLATE
m[0] += self.cn0_shift
return m
def sds(self) -> F64:
s = np.full(len(CNS), self.sigma)
s[0] = max(self.sigma, self.cn0_sigma)
return s
def hwe_weights(f: F64) -> F64:
"""CN weights under Hardy–Weinberg from haplotype copy-number frequencies f_k (k = 0 … 3 copies):
π_c = Σ_{i+j=c} f_i f_j, CN ≥ 5 pooled into 5 [LIT: HWE; OURS: use as the CN mixture prior]."""
pi = np.zeros(len(CNS))
for i in range(len(f)):
for j in range(len(f)):
pi[min(i + j, CN_MAX)] += f[i] * f[j]
return pi
def _hap_update(r: F64, f: F64) -> F64:
"""M-step for f: split each sample's CN responsibilities over the haplotype pairs (i, j) with i + j = c."""
counts = HAP_ALPHA - 1.0
counts = counts.astype(np.float64).copy()
pi = hwe_weights(f)
for i in range(len(f)):
for j in range(len(f)):
c = min(i + j, CN_MAX)
share = f[i] * f[j] / max(pi[c], 1e-300)
w = float(r[:, c].sum()) * share
counts[i] += w
counts[j] += w
counts = np.maximum(counts, 1e-9)
out: F64 = counts / counts.sum()
return out
def _norm_pdf(x: F64, mu: F64, sd: float | F64) -> F64:
sd_ = np.broadcast_to(np.asarray(sd, dtype=np.float64), mu.shape)[None, :]
return np.exp(-0.5 * ((x[:, None] - mu[None, :]) / sd_) ** 2) / (sd_ * math.sqrt(2 * math.pi))
def _em_region(x: F64, gamma: float, delta: float, iters: int, eps0: float = 0.0) -> tuple[RegionFit, float]:
"""EM for one region; returns the fit and its penalised log-likelihood (HWE weights from haplotype copy
frequencies with a Dirichlet prior, Gaussian prior on γ).
CN 0 has its own shift ε₀ ≤ 0 below the template and its own SD, both shrunk towards the template with a prior
weight of ``CN0_PRIOR_N`` samples (on some probe sets CN 0 is far lower than the compressed template, GSTT1)."""
n = len(x)
f = np.array([0.1, 0.8, 0.07, 0.03])
pi = hwe_weights(f)
sd = max(SIGMA_FLOOR, float(np.std(x)) if n > 1 else 0.05)
sd0 = sd
if eps0 < 0: # a start with a far CN-0 cluster: the other components get the spread of the rest
rest = x[x > delta + gamma * TEMPLATE[0] + eps0 / 2]
sd = max(SIGMA_FLOOR, float(np.std(rest)) if len(rest) > 1 else 0.05)
sd0 = sd
g0, gs = GAMMA_PRIOR
ll = -math.inf
for _ in range(iters):
mu = delta + gamma * TEMPLATE
mu[0] += eps0
sds = np.full(len(CNS), sd)
sds[0] = sd0
dens = pi[None, :] * _norm_pdf(x, mu, sds) + 1e-300
tot = dens.sum(axis=1, keepdims=True)
ll = float(np.log(tot).sum() + ((HAP_ALPHA - 1) * np.log(np.maximum(f, 1e-12))).sum()
- 0.5 * ((gamma - g0) / gs) ** 2) # fmt: skip
r = dens / tot
f = _hap_update(r, f)
pi = hwe_weights(f)
# δ and γ: weighted least squares of x on (1, t_c) over CN ≥ 1, γ shrunk towards its prior
rr = r.copy()
rr[:, 0] = 0.0
w = rr.sum(axis=0)
sw, st, stt = float(w.sum()), float((w * TEMPLATE).sum()), float((w * TEMPLATE**2).sum())
sx, stx = float((rr * x[:, None]).sum()), float((rr * x[:, None] * TEMPLATE[None, :]).sum())
lam = sd**2 / gs**2
det = sw * (stt + lam) - st * st
if det > 0:
delta = (sx * (stt + lam) - st * (stx + lam * g0)) / det
gamma = float(np.clip((sw * (stx + lam * g0) - st * sx) / det, *GAMMA_RANGE))
mu = delta + gamma * TEMPLATE
n1 = float(w.sum())
if n1 > 0:
sd = max(SIGMA_FLOOR, math.sqrt(float((rr * (x[:, None] - mu[None, :]) ** 2).sum() / n1)))
r0 = r[:, 0]
n0 = float(r0.sum())
eps0 = min(0.0, float((r0 * (x - mu[0])).sum()) / (n0 + CN0_PRIOR_N))
v0 = float((r0 * (x - mu[0] - eps0) ** 2).sum())
sd0 = max(sd, math.sqrt((v0 + CN0_PRIOR_N * sd**2) / (n0 + CN0_PRIOR_N)))
return RegionFit(float(delta), float(gamma), sd, pi.tolist(), n, float(eps0), float(sd0)), ll
def fit_region(x: F64, iters: int = 200) -> RegionFit:
"""PCN-009 step 4 for one region: the best of EM runs started with each observed cluster labelled CN 0 … 4, each
with CN 0 on the template or on the lowest values."""
g0 = GAMMA_PRIOR[0]
cands = sorted({round(float(q) - g0 * float(t), 4) for q in np.quantile(x, [0.1, 0.3, 0.5, 0.7, 0.9])
for t in TEMPLATE[:5]}) # fmt: skip
lo = float(np.quantile(x, 0.02))
best: tuple[float, float, float] | None = None
for d0 in cands:
for e0 in sorted({0.0, round(min(0.0, lo - (d0 + g0 * float(TEMPLATE[0]))), 4)}):
_, ll = _em_region(x, g0, d0, 80, e0)
if best is None or ll > best[0] + 1e-9:
best = (ll, d0, e0)
assert best is not None
return _em_region(x, g0, best[1], iters, best[2])[0]
def fit_mixture(values: dict[str, F64], iters: int = 200) -> dict[str, RegionFit]:
"""Per-region fits for every region with ≥ 3 finite values."""
out = {}
for name, x in values.items():
x = x[np.isfinite(x)]
if len(x) >= 3:
out[name] = fit_region(x, iters)
return out
def posterior(x: float, fit: RegionFit) -> F64:
"""P(CN = c | x) for c = 0 … 5, with the training weights (floored) as prior."""
pi = np.maximum(np.asarray(fit.weights), PI_FLOOR)
pi = pi / pi.sum()
d = pi * _norm_pdf(np.array([x]), fit.means(), fit.sds())[0]
s = float(d.sum())
out: F64 = d / s if s > 0 else pi
return out
def call_cn(x: float, fit: RegionFit) -> tuple[int, int, F64]:
"""PCN-009 steps 5–6: (CN, QUAL, posterior)."""
p = posterior(x, fit)
cn = int(np.argmax(p))
pmax = float(p[cn])
qual = 60 if pmax >= 1.0 else int(min(60, max(0, round(-10 * math.log10(max(1e-300, 1 - pmax))))))
return cn, qual, p
@@ -0,0 +1,11 @@
# simurg PGx CN regions v1, GRCh38, from UCSC ncbiRefSeq (PCN-001)
chr1 109687816 109693745 GSTM1 GSTM1
chr4 68537172 68576322 UGT2B17 UGT2B17
chr10 133527362 133539123 CYP2E1 CYP2E1
chr16 28605257 28610044 SULT1A1 SULT1A1
chr19 40844772 40845293 CYP2A6.intron.7 CYP2A6
chr19 40849746 40850947 CYP2A6.exon.1 CYP2A6
chr22 42126498 42126752 CYP2D6.exon.9 CYP2D6
chr22 42129185 42129737 CYP2D6.intron.2 CYP2D6
chr22 42130810 42132810 CYP2D6.p5 CYP2D6
chr22_KI270879v1_alt 270313 278446 GSTT1 GSTT1
+510
View File
@@ -0,0 +1,510 @@
"""``simurg pgx copy-number train|call`` — targeted PGx copy number (spec PCN-001 … PCN-017, CLI-031/032, IN-050).
I/O and orchestration only; the numerics live in ``simurg.pgx.cn``.
Deviations (K-028): the per-probe reference is the training cohort's median, stored in the model (not the calling
batch's median, which is not CN 2 in regions with common deletions such as GSTM1); no GC correction step (the region
median over many probes plus the sample scale already removes most of it); GSTT1 needs alt-contig probe coordinates,
which the GSA v4 PGx GRCh38 manifest carries (160 probes on `chr22_KI270879v1_alt`); manifests without them give
`NoCall` for GSTT1.
"""
from __future__ import annotations
import hashlib
import io
import json
import math
import zipfile
from dataclasses import dataclass, field
from datetime import UTC, datetime
from importlib import resources
from pathlib import Path
from typing import Any
import numpy as np
import numpy.typing as npt
import pyarrow as pa
import pyarrow.parquet as pq
from simurg import __version__
from simurg.core.errors import ExitCode, SimurgError
from simurg.core.log import Reporter
from simurg.formats.bpm import read_bpm
from simurg.pgx.cn import CN_MAX, Region, RegionFit, call_cn, fit_mixture, parse_regions
MODEL_SCHEMA = 1
REGIONS_FILE = "pgx_cn_regions.GRCh38.v1.bed"
N_SCALE_PROBES = 20_000
MIN_REGION_PROBES = 10 # PCN-008
MIN_FINITE_FRAC = 0.5 # PCN-002: fewer finite region values → NoCall [OURS]
F32 = npt.NDArray[np.float32]
def default_regions() -> tuple[str, str]:
"""(file name, text) of the packaged region BED (PCN-001)."""
return REGIONS_FILE, resources.files("simurg.pgx.data").joinpath(REGIONS_FILE).read_text(encoding="utf-8")
def _core(chrom: str) -> str:
c = chrom[3:] if chrom.lower().startswith("chr") else chrom
return "X" if c == "XY" else c
@dataclass
class PgxCnModel:
header: dict[str, Any]
regions: list[Region]
regions_text: str
probe_idx: npt.NDArray[np.int64] # manifest indices of all probes used (region probes, then scale probes)
probe_region: list[str] # region name, "" for scale probes
probe_ref: F32 # training reference (median sample-centred LRR)
fits: dict[str, RegionFit] = field(default_factory=dict)
def region_sel(self, name: str) -> npt.NDArray[np.int64]:
return np.flatnonzero(np.asarray(self.probe_region) == name)
def region_probes(
regions: list[Region], chroms: list[str], pos: npt.NDArray[np.int64], masked: set[int]
) -> dict[str, npt.NDArray[np.int64]]:
"""Manifest indices inside each region (1-based position in (start0, end]), mask removed (PCN-008)."""
core = np.array([_core(c) for c in chroms])
out = {}
for r in regions:
sel = np.flatnonzero((core == _core(r.chrom)) & (pos > r.start0) & (pos <= r.end))
out[r.name] = np.array([i for i in sel.tolist() if i not in masked], dtype=np.int64)
return out
def scale_probes(
chroms: list[str], pos: npt.NDArray[np.int64], exclude: npt.NDArray[np.int64], n: int = N_SCALE_PROBES
) -> npt.NDArray[np.int64]:
"""Seeded random autosomal non-target probes for the per-sample scale (PCN-009 step 2)."""
core = np.array([_core(c) for c in chroms])
ok = np.isin(core, [str(i) for i in range(1, 23)]) & (pos > 0)
ok[exclude] = False
cand = np.flatnonzero(ok)
rng = np.random.default_rng(20260929)
return np.sort(rng.choice(cand, min(n, len(cand)), replace=False)).astype(np.int64)
def _centred(lrr: npt.NDArray[np.float32], auto_idx: npt.NDArray[np.int64]) -> npt.NDArray[np.float64]:
x = lrr.astype(np.float64)
x[~np.isfinite(x)] = np.nan
med = float(np.nanmedian(x[auto_idx])) if np.isfinite(x[auto_idx]).any() else 0.0
return x - med
def region_values(d: npt.NDArray[np.float64], model: PgxCnModel) -> dict[str, tuple[float, int]]:
"""Per region: (x, n finite probes). ``d`` = centred LRR at ``model.probe_idx`` minus the reference."""
scale_sel = model.region_sel("")
sc = float(np.nanmedian(d[scale_sel])) if np.isfinite(d[scale_sel]).any() else 0.0
out = {}
for r in model.regions:
sel = model.region_sel(r.name)
v = d[sel]
nf = int(np.isfinite(v).sum())
out[r.name] = (float(np.nanmedian(v)) - sc if nf else math.nan, nf)
return out
# ------------------------------------------------------------------------------------------------ model file
def write_model(path: Path, m: PgxCnModel) -> None:
table = pa.table({"index": m.probe_idx, "region": m.probe_region, "ref": m.probe_ref,
"name": m.header.pop("_names")}) # fmt: skip
buf = io.BytesIO()
pq.write_table(table, buf, compression="zstd")
mixture = {name: {"delta": f.delta, "gamma": f.gamma, "sigma": f.sigma, "weights": f.weights, "n": f.n,
"cn0_shift": f.cn0_shift, "cn0_sigma": f.cn0_sigma} for name, f in m.fits.items()} # fmt: skip
path.parent.mkdir(parents=True, exist_ok=True)
tmp = path.with_suffix(path.suffix + ".part")
with zipfile.ZipFile(tmp, "w", compression=zipfile.ZIP_STORED) as z:
z.writestr("header.json", json.dumps(m.header, indent=1))
z.writestr("probes.parquet", buf.getvalue())
z.writestr("regions.bed", m.regions_text)
z.writestr("mixture.json", json.dumps(mixture, indent=1))
tmp.replace(path)
def read_model(path: Path) -> PgxCnModel:
"""IN-050."""
with zipfile.ZipFile(path) as z:
header = json.loads(z.read("header.json"))
table = pq.read_table(io.BytesIO(z.read("probes.parquet")))
regions_text = z.read("regions.bed").decode()
mixture = json.loads(z.read("mixture.json"))
if int(header.get("schema_version", 0)) < MODEL_SCHEMA:
raise SimurgError("E_MODEL_VERSION", f"{path}: PGx CN model schema {header.get('schema_version')} is too old",
hint="retrain it with simurg pgx copy-number train") # fmt: skip
fits = {
k: RegionFit(
v["delta"],
v["gamma"],
v["sigma"],
list(v["weights"]),
int(v["n"]),
v.get("cn0_shift", 0.0),
v.get("cn0_sigma", 0.0),
)
for k, v in mixture.items()
}
return PgxCnModel(header, parse_regions(regions_text), regions_text,
table.column("index").to_numpy().astype(np.int64), table.column("region").to_pylist(),
table.column("ref").to_numpy().astype(np.float32), fits) # fmt: skip
# ------------------------------------------------------------------------------------------------ train
@dataclass
class PgxTrainOptions:
gtcs: Path
bpm: Path
fasta: Path
out: Path
mask: Path | None = None
regions: Path | None = None
summary: Path | None = None
platform: str = "auto"
min_samples: int = 96
min_passing: int = 90
max_samples: int = 3000
lrr_sd_max: float = 0.20
call_rate_min: float = 0.99
assay_control_min: float = 1.0
force: bool = False
argv: list[str] = field(default_factory=list)
def read_mask_tsv(path: Path | None) -> tuple[set[str], str]:
"""IN-052: ``probe_name<TAB>reason``; returns (names, sha256)."""
if path is None:
return set(), ""
data = path.read_bytes()
names = set()
for ln in data.decode("utf-8").splitlines():
if ln.strip() and not ln.startswith("#"):
names.add(ln.split("\t")[0].strip())
return names, hashlib.sha256(data).hexdigest()
def _summary(path: Path | None) -> dict[str, dict[str, str]]:
if path is None or not path.exists():
return {}
lines = path.read_text(encoding="utf-8").splitlines()
head = lines[0].split("\t")
return {r[0]: dict(zip(head, r, strict=False)) for r in (ln.split("\t") for ln in lines[1:] if ln)}
def _assay_control(row: dict[str, str]) -> float:
for k, v in row.items():
if k.endswith("_norm_r") and k.startswith("TGA_Ctrl"):
try:
return float(v)
except ValueError:
return math.nan
return math.nan
def _probe_set_sha(names: list[str], chroms: list[str], pos: npt.NDArray[np.int64]) -> str:
text = "\n".join(f"{n}\t{c}\t{p}" for n, c, p in zip(names, chroms, pos.tolist(), strict=True))
return hashlib.sha256(text.encode()).hexdigest()
def run_train(o: PgxTrainOptions, rep: Reporter) -> int:
"""``simurg pgx copy-number train`` (CLI-032, PCN-005)."""
from simurg.core.atomic import check_outputs_absent
from simurg.formats.discovery import discover_gtcs
from simurg.formats.fasta import Genome
from simurg.formats.gtc import read_gtc
from simurg.pipeline.qc import write_tsv
man = read_bpm(o.bpm)
out_model = o.out / f"{o.bpm.stem}.simurg-pgxcn"
check_outputs_absent([out_model, o.out / "train_exclusions.tsv"], force=o.force)
genome = Genome(o.fasta)
if genome.build not in ("GRCh38", "unknown", ""):
raise SimurgError("E_BUILD_MISMATCH", f"PGx CN supports GRCh38 only (FASTA is {genome.build})") # PCN-015
reg_name, reg_text = (o.regions.name, o.regions.read_text()) if o.regions else default_regions()
regions = parse_regions(reg_text)
mask_path = o.mask or (
o.bpm.with_name(o.bpm.stem + ".mask.tsv") if (o.bpm.with_name(o.bpm.stem + ".mask.tsv")).exists() else None
)
mask_names, mask_sha = read_mask_tsv(mask_path)
masked = {i for i, n in enumerate(man.names) if n in mask_names}
pos = man.map_infos.astype(np.int64)
rp = region_probes(regions, list(man.chroms), pos, masked)
for r in regions:
if len(rp[r.name]) < MIN_REGION_PROBES:
rep.warn("W_PGX_REGION_FEW_PROBES", f"region {r.name}: {len(rp[r.name])} unmasked probe(s) on "
f"{man.manifest_name}; it will be NoCall") # fmt: skip
reg_idx = np.concatenate([rp[r.name] for r in regions]) if regions else np.zeros(0, np.int64)
sc_idx = scale_probes(list(man.chroms), pos, reg_idx)
probe_idx = np.concatenate([reg_idx, sc_idx])
probe_region = [r.name for r in regions for _ in range(len(rp[r.name]))] + [""] * len(sc_idx)
gtcs = discover_gtcs(o.gtcs)
keys = sorted(gtcs)
if len(keys) > o.max_samples:
rng = np.random.default_rng(42)
keys = sorted(rng.choice(keys, o.max_samples, replace=False).tolist())
rep.info(f"training on a seeded random subset of {o.max_samples} of {len(gtcs)} samples (PCN-005)")
summ = _summary(o.summary or (o.gtcs / "sample_summary.tsv"))
rows = np.full((len(keys), len(probe_idx)), np.nan, dtype=np.float32)
exclusions: list[dict[str, str]] = []
passing = np.zeros(len(keys), dtype=np.bool_)
cluster_files = set()
for i, k in enumerate(keys):
g = read_gtc(gtcs[k])
if len(g.logr_ratios) != man.num_loci:
raise SimurgError("E_MODEL_MANIFEST_MISMATCH", f"{gtcs[k]} has {len(g.logr_ratios)} loci, the manifest "
f"{man.num_loci}") # fmt: skip
cluster_files.add(g.cluster_file)
rows[i] = _centred(g.logr_ratios, sc_idx)[probe_idx]
reasons = []
if not g.logr_dev <= o.lrr_sd_max:
reasons.append(f"logr_dev {g.logr_dev:.3f} > {o.lrr_sd_max}")
if not g.call_rate >= o.call_rate_min:
reasons.append(f"call_rate {g.call_rate:.4f} < {o.call_rate_min}")
ac = _assay_control(summ.get(k, {}))
if math.isfinite(ac) and ac < o.assay_control_min:
reasons.append(f"assay_control_norm_r {ac:.3f} < {o.assay_control_min}")
if reasons:
exclusions.append({"sample_id": k, "reason": "; ".join(reasons)})
else:
passing[i] = True
n_pass = int(passing.sum())
if len(keys) < o.min_samples or n_pass < o.min_passing:
raise SimurgError("E_PGX_TRAIN_TOO_SMALL", f"{n_pass} of {len(keys)} sample(s) passed QC; training requires "
f"at least {o.min_samples} samples of which {o.min_passing} pass",
hint="lower --min-samples/--min-passing only for research use") # fmt: skip
ref = np.nanmedian(rows[passing], axis=0)
d = rows - ref
sc_sel = np.flatnonzero(np.asarray(probe_region) == "")
sc = np.nanmedian(d[:, sc_sel], axis=1)
mad = float(np.nanmedian(np.abs(sc[passing] - np.nanmedian(sc[passing])))) * 1.4826 or 1e-9
z = np.abs(sc - np.nanmedian(sc[passing])) / mad
use = passing & (z <= 5) # PCN-009: outlier samples out of the reference
for i in np.flatnonzero(passing & ~use).tolist():
exclusions.append({"sample_id": keys[i], "reason": f"genome-wide NR outlier (|z| = {z[i]:.1f} > 5)"})
if (~use & passing).any():
ref = np.nanmedian(rows[use], axis=0)
d = rows - ref
model = PgxCnModel({}, regions, reg_text, probe_idx, probe_region, ref.astype(np.float32))
values = {}
for r in regions:
if len(rp[r.name]) < MIN_REGION_PROBES:
continue
values[r.name] = np.array([region_values(d[i], model)[r.name][0] for i in np.flatnonzero(use)])
model.fits = fit_mixture(values)
csv_fmt = "unknown" if o.platform == "auto" else o.platform
model.header = {
"schema": "simurg.pgx_cn_model", "schema_version": MODEL_SCHEMA, "software_version": __version__,
"manifest_name": man.manifest_name, "manifest_revision": getattr(man, "version", None),
"probe_set_sha256": _probe_set_sha(man.names, list(man.chroms), pos),
"n_probes": man.num_loci,
"cluster_file_name": sorted(cluster_files)[0] if len(cluster_files) == 1 else sorted(cluster_files),
"genome_build": "GRCh38", "fasta_contigs_sha256": genome.sha256, "platform": csv_fmt,
"mask_sha256": mask_sha, "regions_bed_version": reg_name, "n_training_samples": int(use.sum()),
"training_exclusions": len(exclusions), "created": datetime.now(UTC).strftime("%Y-%m-%dT%H:%M:%SZ"),
"regions_few_probes": [r.name for r in regions if len(rp[r.name]) < MIN_REGION_PROBES],
"model": "mu_c = delta_r + gamma_r * log2((c + 1) / 3), c = 0..5 (simurg.pgx.cn)",
"_names": [man.names[i] for i in probe_idx.tolist()],
} # fmt: skip
o.out.mkdir(parents=True, exist_ok=True)
write_model(out_model, model)
write_tsv(o.out / "train_exclusions.tsv", ["sample_id", "reason"], exclusions)
for name, f in model.fits.items():
rep.info(f"{name}: CN2 at {f.delta:+.3f}, gamma {f.gamma:.3f}, sd {f.sigma:.3f}, "
f"weights {' '.join(f'{w:.2f}' for w in f.weights)}") # fmt: skip
rep.info(f"PGx CN model from {int(use.sum())} sample(s) written to {out_model}")
return ExitCode.OK
# ------------------------------------------------------------------------------------------------ call
@dataclass
class PgxCallOptions:
gtcs: Path
model: Path
fasta: Path
out: Path
min_batch: int = 24
min_passing: int = 22
recommended_passing: int = 96
lrr_sd_max: float = 0.20
cluster_mismatch: str = "error"
bgzip: bool = True
force: bool = False
argv: list[str] = field(default_factory=list)
def _vcf_header(sample: str, contigs: list[tuple[str, int]], meta: dict[str, str]) -> str:
h = ["##fileformat=VCFv4.3", f"##source=SimurgArray {__version__} pgx-cn", f"##reference=file://{meta['fasta']}",
"##simurg_genome_build=GRCh38", f"##simurg_manifest={meta['manifest']}",
f"##simurg_pgx_cn_model={meta['model']}", f"##simurg_batch={meta['batch']}",
f"##OverallPloidy={meta['ploidy']}", "##GCCorrect=false"] # fmt: skip
h += [f"##contig=<ID={c},length={n}>" for c, n in contigs]
h += ['##ALT=<ID=CNV,Description="Copy number variable region (targeted genotype)">',
'##FILTER=<ID=PASS,Description="All filters passed">',
'##FILTER=<ID=Q7,Description="Quality below 7">',
'##FILTER=<ID=SampleQuality,Description="Sample LRR SD above 0.2">',
'##FILTER=<ID=NoCall,Description="Region could not be assessed">',
'##INFO=<ID=END,Number=1,Type=Integer,Description="Last base of the region (1-based, inclusive)">',
'##INFO=<ID=SVTYPE,Number=1,Type=String,Description="CNV">',
'##INFO=<ID=CNVLEN,Number=1,Type=Integer,Description="Region length in bp">',
'##INFO=<ID=PROBE,Number=1,Type=Integer,Description="Number of probes used for the region">',
'##INFO=<ID=REGION,Number=1,Type=String,Description="Target region name">',
'##INFO=<ID=GENE,Number=1,Type=String,Description="Gene symbol">',
'##FORMAT=<ID=CN,Number=1,Type=Integer,Description="Copy number; 5 means 5 or more">',
'##FORMAT=<ID=NR,Number=1,Type=Float,Description="Aggregated normalized intensity (about CN/2)">',
"\t".join(["#CHROM", "POS", "ID", "REF", "ALT", "QUAL", "FILTER", "INFO", "FORMAT", sample])] # fmt: skip
return "\n".join(h) + "\n"
def call_sample_regions(d: npt.NDArray[np.float64], model: PgxCnModel, passing: bool) -> list[dict[str, Any]]:
"""PCN-002/009: one result per region."""
vals = region_values(d, model)
out = []
for r in model.regions:
x, nf = vals[r.name]
n_all = len(model.region_sel(r.name))
fit = model.fits.get(r.name)
filters: list[str] = []
if fit is None or n_all < MIN_REGION_PROBES or nf < MIN_FINITE_FRAC * n_all or not math.isfinite(x):
cn, qual, nr = None, 0, math.nan
filters.append("NoCall")
else:
cn, qual, _ = call_cn(x, fit)
# NR ≈ CN / 2: the continuous CN that the region's mixture means imply for x [OURS]
nr = max(0.0, 3.0 * 2.0 ** ((x - fit.delta) / fit.gamma) - 1.0) / 2.0 if cn else 0.0
if qual < 7:
filters.append("Q7")
if not passing:
filters.append("SampleQuality")
out.append(
{"region": r, "cn": cn, "qual": qual, "nr": nr, "x": x, "probes": nf, "filters": filters or ["PASS"]}
)
return out
def run_call(o: PgxCallOptions, rep: Reporter) -> int:
"""``simurg pgx copy-number call`` (CLI-031)."""
from simurg.core.atomic import atomic_write_text, check_outputs_absent
from simurg.formats.discovery import discover_gtcs
from simurg.formats.fasta import ContigMap, Genome
from simurg.formats.gtc import read_gtc
from simurg.pipeline.cn import write_vcf
from simurg.pipeline.qc import write_tsv
model = read_model(o.model)
genome = Genome(o.fasta)
if genome.build not in ("GRCh38", "unknown", ""):
raise SimurgError("E_BUILD_MISMATCH", f"PGx CN supports GRCh38 only (FASTA is {genome.build})")
cmap = ContigMap([e.name for e in genome.entries], "auto")
gtcs = discover_gtcs(o.gtcs)
ext = ".pgx_cn.vcf.gz" if o.bgzip else ".pgx_cn.vcf"
check_outputs_absent([o.out / f"{k}{ext}" for k in gtcs] + [o.out / "pgx_cn_batch_summary.tsv"], force=o.force)
n_probes = int(model.header["n_probes"])
mcl = model.header.get("cluster_file_name")
# pass 1: batch QC from GTC headers (PCN-004)
heads = {}
for k, p in gtcs.items():
g = read_gtc(p)
heads[k] = (g.logr_dev, g.sample_name or k, g.cluster_file, len(g.logr_ratios))
n_pass = sum(1 for v in heads.values() if v[0] < o.lrr_sd_max)
if len(heads) < o.min_batch or n_pass < o.min_passing:
raise SimurgError("E_PGX_BATCH_TOO_SMALL", f"{n_pass} sample(s) passed QC. Requires at least "
f"{max(o.min_passing, 1)} samples to proceed.") # fmt: skip
if n_pass < o.recommended_passing:
rep.warn("W_PGX_BATCH_BELOW_RECOMMENDED", f"{n_pass} passing sample(s); {o.recommended_passing} recommended")
for k, (_, _, cf, nl) in heads.items():
if nl != n_probes:
raise SimurgError("E_MODEL_MANIFEST_MISMATCH", f"{k} has {nl} loci, the model's manifest {n_probes}")
if isinstance(mcl, str) and cf and cf != mcl:
msg = f"Cluster file ({cf}) differs from the CN model's cluster file ({mcl})"
if o.cluster_mismatch == "error":
raise SimurgError("E_MODEL_CLUSTER_MISMATCH", msg, hint="retrain the model (PCN-005)")
rep.warn("W_MODEL_CLUSTER_MISMATCH", msg)
break
contigs: list[tuple[str, int]] = []
contig_name: dict[str, str | None] = {}
for r in model.regions:
if r.chrom in contig_name:
continue
c = cmap.get(r.chrom)
contig_name[r.chrom] = c
if c is None:
rep.warn("W_PGX_NO_ALT_CONTIG", f"{r.chrom} is not in {o.fasta.name}; REF written as N (IN-062)")
contigs.append((r.chrom, genome.contig_length(c) if c else r.end))
order = {c: i for i, (c, _) in enumerate(contigs)}
model_sha = hashlib.sha256(o.model.read_bytes()).hexdigest()
mh = model.header
meta = {
"fasta": str(o.fasta.resolve()),
"manifest": f"{mh['manifest_name']};sha256={mh['probe_set_sha256']}",
"model": f"{o.model.name};sha256={model_sha};cluster_file={mcl};regions={mh['regions_bed_version']}",
"batch": f"n_samples={len(heads)};n_passing={n_pass};lrr_sd_max={o.lrr_sd_max}",
}
centre_idx = model.probe_idx[model.region_sel("")] # the scale probes (autosomal), as in training
o.out.mkdir(parents=True, exist_ok=True)
srows, per_region = [], {r.name: {"n_gain": 0, "n_loss": 0, "n_neutral": 0, "n_nocall": 0, "n_filtered": 0}
for r in model.regions} # fmt: skip
failed = 0
for k, p in gtcs.items():
try:
g = read_gtc(p)
d = _centred(g.logr_ratios, centre_idx)[model.probe_idx] - model.probe_ref
passing = heads[k][0] < o.lrr_sd_max
res = call_sample_regions(d, model, passing)
except SimurgError:
raise
except Exception as e: # PCN: one sample never stops the batch
failed += 1
rep.error("E_PGX_CN_FAILED", f"{k}: {e}", sample_id=k)
continue
called = [(x["region"], x["cn"]) for x in res if x["cn"] is not None]
tot = sum(r.end - r.start0 for r, _ in called)
ploidy = sum((r.end - r.start0) * min(cn, CN_MAX) for r, cn in called) / tot if tot else math.nan
meta_s = {**meta, "ploidy": f"{ploidy:.3f}" if math.isfinite(ploidy) else "."}
recs = []
for x in sorted(res, key=lambda y: (order[y["region"].chrom], y["region"].start0)):
r = x["region"]
c = contig_name.get(r.chrom)
ref = (genome.fetch(c, r.start0 - 1, r.start0) or "N").upper() if c and r.start0 >= 1 else "N"
s, e_ = r.start0 + 1, r.end
cn = "." if x["cn"] is None else str(x["cn"])
nr = "." if not math.isfinite(x["nr"]) else f"{x['nr']:.6f}"
line = "\t".join([r.chrom, str(s - 1), f"CNV:{r.name}:{r.chrom}:{s}-{e_}", ref, "<CNV>", str(x["qual"]),
";".join(x["filters"]), f"END={e_};SVTYPE=CNV;CNVLEN={e_ - s + 1};PROBE={x['probes']};"
f"REGION={r.name};GENE={r.gene}", "CN:NR", f"{cn}:{nr}"]) # fmt: skip
recs.append((r.chrom, s - 2, e_, line))
pr = per_region[r.name]
if x["cn"] is None:
pr["n_nocall"] += 1
elif x["cn"] > 2:
pr["n_gain"] += 1
elif x["cn"] < 2:
pr["n_loss"] += 1
else:
pr["n_neutral"] += 1
pr["n_filtered"] += x["filters"] != ["PASS"]
write_vcf(o.out / f"{k}{ext}", _vcf_header(heads[k][1], contigs, meta_s), recs, o.bgzip)
srows.append({"sample_id": k, "sample_name": heads[k][1], "sample_folder": str(Path(p).parent),
"logr_dev": f"{heads[k][0]:.6f}", "passed_batch_qc": str(passing).lower(),
"used_in_reference": "false",
**{f"cn_{x['region'].name}": ("." if x["cn"] is None else x["cn"]) for x in res},
"flags": ""}) # fmt: skip
cols = ["region", "gene", "n_gain", "n_loss", "n_neutral", "n_nocall", "n_filtered"]
rows: list[dict[str, Any]] = [
{"region": r.name, "gene": r.gene, **per_region[r.name]} for r in sorted(model.regions, key=lambda r: r.name)
]
totals = {c: sum(per_region[r.name][c] for r in model.regions) for c in cols[2:]}
rows.append({"region": "ALL", "gene": "", **totals})
write_tsv(o.out / "pgx_cn_batch_summary.tsv", cols, rows)
scols = ["sample_id", "sample_name", "sample_folder", "logr_dev", "passed_batch_qc", "used_in_reference"]
scols += [f"cn_{r.name}" for r in model.regions] + ["flags"]
write_tsv(o.out / "pgx_cn_sample_summary.tsv", scols, srows)
run = {"software": f"SimurgArray {__version__}", **meta, "samples": len(gtcs), "failed": failed}
atomic_write_text(o.out / "pgx_cn_run.json", json.dumps(run, indent=1))
rep.info(f"PGx CN: {len(srows)}/{len(gtcs)} sample(s) called")
if failed == 0:
return ExitCode.OK
return ExitCode.PARTIAL if srows else ExitCode.ALL_FAILED
+145
View File
@@ -0,0 +1,145 @@
"""Targeted PGx copy number (spec PCN-001 … PCN-017, CLI-031/032, IN-050)."""
from __future__ import annotations
import gzip
from pathlib import Path
import numpy as np
from typer.testing import CliRunner
from simurg.cli.main import app
from simurg.formats.gtc import Gtc, write_gtc
from simurg.pgx.cn import TEMPLATE, call_cn, fit_region, hwe_weights, parse_regions
from simurg.pipeline.pgx_cn import default_regions
from simurg.synth.files import SynthLocus, encode_bpm
def _cluster_values(rng: np.random.Generator, cns: np.ndarray, delta: float, gamma: float, sd: float,
cn0_drop: float = 0.0) -> np.ndarray: # fmt: skip
x = delta + gamma * TEMPLATE[cns] + rng.normal(0, sd, len(cns))
return np.where(cns == 0, x + cn0_drop, x)
def _hwe_cns(rng: np.random.Generator, n: int, f: list[float]) -> np.ndarray:
return rng.choice(len(f), n, p=f) + rng.choice(len(f), n, p=f)
def test_hwe_weights() -> None: # PCN-009 [OURS]
w = hwe_weights(np.array([0.3, 0.6, 0.1, 0.0]))
assert abs(w.sum() - 1) < 1e-12
assert np.allclose(w[:4], [0.09, 0.36, 0.06 + 0.36, 0.12])
def test_labels_identified() -> None: # PCN-009: GSTM1-like, GSTT1-like and one-cluster regions
rng = np.random.default_rng(1)
cns = _hwe_cns(rng, 60, [0.7, 0.3]) # common deletion: mostly CN 0 and 1
x = _cluster_values(rng, cns, 0.1, 0.22, 0.02)
f = fit_region(x)
assert np.mean([call_cn(v, f)[0] for v in x] == cns) > 0.95
cns = _hwe_cns(rng, 60, [0.45, 0.55])
x = _cluster_values(rng, cns, 0.3, 0.55, 0.02, cn0_drop=-1.2) # CN 0 far below the template
f = fit_region(x)
assert f.cn0_shift < -0.5
assert np.mean([call_cn(v, f)[0] for v in x] == cns) > 0.95
cns = np.full(40, 2)
cns[:2] = 1
x = _cluster_values(rng, cns, -0.05, 0.25, 0.015)
f = fit_region(x)
assert [call_cn(v, f)[0] for v in x] == cns.tolist()
def test_packaged_regions() -> None: # PCN-001
name, text = default_regions()
regs = parse_regions(text)
assert name.endswith(".v1.bed") and len(regs) == 10
assert {r.gene for r in regs} == {"CYP2D6", "CYP2A6", "GSTM1", "GSTT1", "UGT2B17", "SULT1A1", "CYP2E1"}
assert next(r for r in regs if r.gene == "GSTT1").chrom == "chr22_KI270879v1_alt"
def _dataset(tmp: Path, n_samples: int = 40) -> tuple[Path, Path, Path, Path]:
rng = np.random.default_rng(5)
loci = []
for c in ("1", "2"):
for k in range(1500):
loci.append(SynthLocus(f"cnv{c}_{k}", c, 2000 * (k + 1), "[A/G]", len(loci) + 1, 0, 0))
bpm = tmp / "m.bpm"
bpm.write_bytes(encode_bpm(loci))
fa = tmp / "g.fa"
body = {c: "".join(rng.choice(list("ACGT"), 1500 * 2000 + 10_000)) for c in ("1", "2")}
fa.write_text("".join(f">{c}\n" + "\n".join(s[i : i + 60] for i in range(0, len(s), 60)) + "\n"
for c, s in body.items())) # fmt: skip
regions = tmp / "regions.bed"
regions.write_text("# test regions\n1\t100000\t160000\tGENEA\tGENEA\n2\t300000\t340000\tGENEB\tGENEB\n"
"3\t1000\t5000\tGENEC\tGENEC\n") # fmt: skip
n = len(loci)
pos = np.array([lc.pos for lc in loci])
chrom = np.array([lc.chrom for lc in loci])
in_a = (chrom == "1") & (pos > 100_000) & (pos <= 160_000)
in_b = (chrom == "2") & (pos > 300_000) & (pos <= 340_000)
cn_a = _hwe_cns(rng, n_samples, [0.5, 0.5])
gtcs = tmp / "gtcs"
truth = {}
for i in range(n_samples):
sid = f"2000000000{i:02d}_R01C01"
lrr = rng.normal(0, 0.08, n)
lrr[in_a] += 0.24 * TEMPLATE[cn_a[i]] + (-0.2 if cn_a[i] == 0 else 0.0)
cn_b = 3 if i == 0 else 2
lrr[in_b] += 0.25 * TEMPLATE[cn_b]
truth[sid] = (int(cn_a[i]), cn_b)
g = Gtc(num_snps=n, genotypes=np.ones(n, np.uint8), genotype_scores=np.full(n, 0.9, np.float32),
b_allele_freqs=np.zeros(n, np.float32), logr_ratios=lrr.astype(np.float32),
raw_x=np.full(n, 1000, np.uint16), raw_y=np.full(n, 1000, np.uint16),
base_calls=np.zeros((n, 2), np.uint8), gender=b"F", call_rate=0.995, logr_dev=0.12,
cluster_file="c.egt") # fmt: skip
(gtcs / sid).mkdir(parents=True)
write_gtc(g, gtcs / sid / f"{sid}.gtc")
(tmp / "truth.txt").write_text("\n".join(f"{k}\t{a}\t{b}" for k, (a, b) in truth.items()))
return bpm, fa, gtcs, regions
def test_pgx_cn_train_and_call(tmp_path: Path) -> None: # CLI-031/032, PCN-002/004/011/012/013, IN-050
bpm, fa, gtcs, regions = _dataset(tmp_path)
run = CliRunner()
r = run.invoke(app, ["pgx", "copy-number", "train", "--gtcs", str(gtcs), "--bpm", str(bpm), "--fasta", str(fa),
"--out", str(tmp_path / "model"), "--regions", str(regions), "--quiet"]) # fmt: skip
assert r.exit_code == 2 and "E_PGX_TRAIN_TOO_SMALL" in r.output # 40 < 96 (PCN-005)
r = run.invoke(app, ["pgx", "copy-number", "train", "--gtcs", str(gtcs), "--bpm", str(bpm), "--fasta", str(fa),
"--out", str(tmp_path / "model"), "--regions", str(regions), "--min-samples", "30",
"--min-passing", "30", "--quiet"]) # fmt: skip
assert r.exit_code == 0, r.output
model = tmp_path / "model" / "m.simurg-pgxcn"
assert model.exists() and (tmp_path / "model" / "train_exclusions.tsv").exists()
out = tmp_path / "call"
r = run.invoke(app, ["pgx", "copy-number", "call", "--gtcs", str(gtcs), "--model", str(model), "--fasta", str(fa),
"--out", str(out), "--min-batch", "50", "--quiet"]) # fmt: skip
assert r.exit_code == 2 and "E_PGX_BATCH_TOO_SMALL" in r.output
r = run.invoke(app, ["pgx", "copy-number", "call", "--gtcs", str(gtcs), "--model", str(model), "--fasta", str(fa),
"--out", str(out), "--quiet"]) # fmt: skip
assert r.exit_code == 0, r.output
truth = {ln.split("\t")[0]: (int(ln.split("\t")[1]), int(ln.split("\t")[2]))
for ln in (tmp_path / "truth.txt").read_text().splitlines()} # fmt: skip
ok_a = ok_b = 0
for sid, (a, b) in truth.items():
with gzip.open(out / f"{sid}.pgx_cn.vcf.gz", "rt") as fh:
text = fh.read()
assert (out / f"{sid}.pgx_cn.vcf.gz.tbi").exists()
recs = [ln.split("\t") for ln in text.splitlines() if not ln.startswith("#")]
assert [x[7].split("REGION=")[1].split(";")[0] for x in recs] == ["GENEA", "GENEB", "GENEC"] # PCN-002
for x in recs: # PCN-011: POS = S − 1, ID coordinates = POS + 1 … END
end = int(x[7].split("END=")[1].split(";")[0])
s_, e_ = x[2].split(":")[-1].split("-")
assert int(s_) == int(x[1]) + 1 and int(e_) == end and x[8] == "CN:NR"
genec = recs[2]
assert genec[6] == "NoCall" and genec[9].startswith(".:")
ok_a += int(recs[0][9].split(":")[0]) == a
ok_b += int(recs[1][9].split(":")[0]) == b
assert ok_a >= 0.95 * len(truth) and ok_b >= 0.9 * len(truth), (ok_a, ok_b)
rows = [ln.split("\t") for ln in (out / "pgx_cn_batch_summary.tsv").read_text().splitlines()]
assert rows[0] == ["region", "gene", "n_gain", "n_loss", "n_neutral", "n_nocall", "n_filtered"]
assert [r_[0] for r_ in rows[1:]] == ["GENEA", "GENEB", "GENEC", "ALL"]
for r_ in rows[1:-1]: # PCN-012: the four counts add up to N
assert sum(int(v) for v in r_[2:6]) == len(truth)
head = (out / "pgx_cn_sample_summary.tsv").read_text().splitlines()[0].split("\t")
assert head[:6] == ["sample_id", "sample_name", "sample_folder", "logr_dev", "passed_batch_qc",
"used_in_reference"] and head[-1] == "flags" # fmt: skip