Commit Graph
6 Commits
Author SHA1 Message Date
kemal 7904399161 P5: cloud task for cohort QC validation; configurable gp3 throughput; batched contamination map
- scripts/cloud/tasks/p5.sh: P4 preset re-check, genotype (v5) with QC, validate_p5_large.py
- scripts/validate_p5_large.py: identify samples via het pattern vs 1000G VCF; sex, parent-offspring,
  Mendel, contamination/control distributions
- trial.sh: DISK_MBPS / DISK_IOPS
- qc/contamination: bounded map batches (executor.map submits eagerly)
2026-09-28 09:10:02 +01:00
kemal 7032074bfe P4: unit tests for calling/BAF/LRR/sex/CLI; vendor validation script
- tests/unit/test_genotype.py: fuzzy memberships incl. NaN rules, calls
  at cluster centres, intensity threshold, BAF/LRR interpolation and
  NaN pattern, LogR-dev locus set, sex rules v1/v2, TOP-strand base
  calls, invariant dates, end-to-end `simurg genotype` (summary schema,
  pass-through columns, thread/layout determinism, overwrite refusal)
- tests/oracle/test_p4_genotype.py: 36 GSA samples x 2 versions
  byte-identical to the oracle
- scripts/validate_p4_large.py + cloud task p4: compare every vendor
  GTC field by field and as whole files; reports which sex rule matches
2026-09-28 00:10:47 +01:00
kemal e308253039 P3: vectorised nearest-neighbour search; task-driven cloud trials
- fit.py: exact and binned nearest-neighbour searches use one distance
  matrix per searched bin (first-minimum rule kept); fit 2.6 s -> 1.4 s
  per GSA sample, still bit-identical to the oracle
- scripts/validate_p3_large.py: refits 1.2.0 and 1.1.2 transforms for
  every vendor GTC and reports ULP distances per field
- cloud trial runs scripts/cloud/tasks/<TASK>.sh (TASK=p3 default;
  p1p2 = previous validation + ingest)
2026-09-27 22:57:03 +01:00
kemal 17bd0e8f78 P2 validation: oracle test and large-scale raw X/Y check
- tests/oracle/test_p2_raw_xy.py: IDAT-assembled raw X/Y and controls
  equal the arrays in oracle GTCs
- validate_p1_large.py: phase-P2 comparison against vendor GTCs with
  diagnostics (channel swap, bead-count disagreements)
- cloud trial: also runs `simurg ingest` on all samples (time/memory),
  disk 250 GB
2026-09-27 20:57:42 +01:00
kemal 795dc6d8d8 cloud trial: set HOME/PATH for cloud-init, unit tests non-fatal
The first run stopped after 1 minute (HOME unbound under set -u) and
terminated itself as designed.
2026-09-27 19:36:49 +01:00
kemal 3bfb7b5757 Add self-destructing AWS trial and large-scale P1 validation script
scripts/cloud/trial.sh launches one spot instance that validates the
parsers on the 1000 Genomes Omni2.5 set (2141 samples, vendor GTCs)
streamed from the public S3 bucket, uploads the summary and terminates
itself. Cost guards: spot price cap, hard time limit (shutdown =
terminate), EXIT-trap shutdown, CloudWatch idle alarm that terminates
from outside, disk deleted with the instance, monthly budget alert, and
an `audit` command listing anything still billable.

scripts/validate_p1_large.py compares every GTC with BeadArrayFiles,
checks byte-identical round-trips, IDAT pairs and BPM/EGT/CSV.
2026-09-27 19:30:27 +01:00