kemal/SimurgArray
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
61 Commits 1 Branch 3 Tags
f1a30aa10471768bccf61e0daf770039394423f6
Commit Graph
61 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
kemalandClaude Opus 5.5 2e75345a30 P2 validation: oracle test and large-scale raw X/Y check
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
- tests/oracle/test_p2_raw_xy.py: IDAT-assembled raw X/Y and controls
  equal the arrays in oracle GTCs
- validate_p1_large.py: phase-P2 comparison against vendor GTCs with
  diagnostics (channel swap, bead-count disagreements)
- cloud trial: also runs `simurg ingest` on all samples (time/memory),
  disk 250 GB

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 20:57:42 +01:00
kemalandClaude Opus 5.5 a4faead6d7 P2 (WIP): raw X/Y assembly, canonical Parquet model, simurg ingest
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
- signal/raw_xy.py: vectorised Infinium I/II raw X/Y and control assembly
  (plan §7.1) with a per-chip address-map cache; IN-027 coverage gate
- core/schemas.py: Arrow schemas (loci, sample_intensity, controls),
  deterministic atomic Parquet writing
- pipeline/ingest.py + CLI: discovery, sheet, IN-010 naming, overwrite
  check before any work, spawn-based workers, per-sample isolation,
  --resume by input hashes, run manifest, exit codes 0/3/4
- matches gtc2vcf raw X/Y and controls exactly on 18 GSA samples;
  outputs byte-identical across thread counts and layouts

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 20:54:06 +01:00
kemalandClaude Opus 5.5 854fe132cc P1 validated on 1000 Genomes Omni2.5 (2141 vendor GTCs + IDATs)
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
All 2141 vendor-written GTC v3 files match BeadArrayFiles field by field
and re-encode byte-identically; all IDAT pairs parse; the 2.45M-locus BPM,
dbv-7 EGT and CSV manifest agree with BeadArrayFiles and each other.
Run on a self-destructing AWS spot instance (42 min, ~$0.20).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 20:22:35 +01:00
kemalandClaude Opus 5.5 f52313fc93 cloud trial: set HOME/PATH for cloud-init, unit tests non-fatal
ci / test (3.12) (push) Failing after 8s
Details
ci / test (3.13) (push) Failing after 10s
Details 
The first run stopped after 1 minute (HOME unbound under set -u) and
terminated itself as designed.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 19:36:49 +01:00
kemalandClaude Opus 5.5 150650424c Lint fixes in validate_p1_large.py
ci / test (3.12) (push) Failing after 10s
Details
ci / test (3.13) (push) Failing after 12s
Details 
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 19:30:37 +01:00
kemalandClaude Opus 5.5 a9aa6dc080 Add self-destructing AWS trial and large-scale P1 validation script
ci / test (3.12) (push) Failing after 9s
Details
ci / test (3.13) (push) Failing after 13s
Details 
scripts/cloud/trial.sh launches one spot instance that validates the
parsers on the 1000 Genomes Omni2.5 set (2141 samples, vendor GTCs)
streamed from the public S3 bucket, uploads the summary and terminates
itself. Cost guards: spot price cap, hard time limit (shutdown =
terminate), EXIT-trap shutdown, CloudWatch idle alarm that terminates
from outside, disk deleted with the instance, monthly budget alert, and
an `audit` command listing anything still billable.

scripts/validate_p1_large.py compares every GTC with BeadArrayFiles,
checks byte-identical round-trips, IDAT pairs and BPM/EGT/CSV.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 19:30:27 +01:00
kemalandClaude Opus 5.5 4357d3deef P1: file parsers, sample sheet, discovery, CLI inspect
ci / test (3.12) (push) Failing after 11s
Details
ci / test (3.13) (push) Failing after 12s
Details 
Readers for IDAT v3, BPM (locus entries v4-v8), CSV manifest, EGT
(data blocks 5-9) and GTC v3-v5 (plus a layout-preserving GTC writer);
sample-sheet validation (IN-001..IN-014), IDAT discovery and pairing
(IN-020..IN-022), message catalogue and exit codes (CLI-014/016),
JSON Lines logging (CLI-007/008), atomic writes (CLI-011), run manifest,
and `simurg inspect`.

Validation (docs/validation/P1.md): GTC, BPM and EGT readers agree
exactly with Illumina BeadArrayFiles on GSA v4 / GDA v1 files; GTCs
round-trip byte-identically; BPM and CSV manifests are consistent.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 16:56:39 +01:00
kemalandClaude Opus 5.5 f10c651f0f P0: project setup (packaging, CI, licences, data/oracle scripts) 
- pyproject (uv/hatch, ruff, mypy strict, pytest markers), MIT LICENSE,
  THIRD_PARTY_NOTICES with verbatim BSD-2 (BeadArrayFiles) and MIT (gtc2vcf)
- CI workflow for Gitea/GitHub Actions: lint, type check, unit tests
- fetch_data.sh p0 now fetches the GSA v4 PGx demo IDATs (the GDA "Demo 36"
  bundle is a GenomeStudio project), unpacks archives, fixes pipefail bug
- make_oracle_gtcs.sh builds bcftools + gtc2vcf and writes oracle GTCs
- plan §17: corrected P0 dataset table

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 16:56:39 +01:00
kemalandClaude Opus 5.5 0861b27c2b Add sanitized Phase-0 research archive under docs/research 
Research notes R1-R4, documentation-study files A-E (source of the
functional spec's `src:` tags) and PGx coverage CSVs, rewritten to use
neutral naming and current project decisions (plan §2).

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 16:27:34 +01:00
kemalandClaude Opus 5.5 6449440492 Add functional specification (416 items) and link it from plan/docs 
SIMURGARRAY_FUNCTIONAL_SPEC.md defines externally visible behaviour:
CLI, inputs, genotyping outputs, QC and report, cytogenetics CNV/LOH,
targeted PGx copy number, star alleles, methylation control QC, edge-case
catalogue, performance targets, open questions and a traceability table.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 16:13:40 +01:00
kemalandClaude Opus 5.5 ee4ebaaa1e SimurgArray: master plan, repo scaffolding and data/ref fetch scripts 
Phase 0 deliverables: full technical specification and roadmap
(SIMURGARRAY_MASTER_PLAN.md), CLAUDE.md, bootstrap script for reference
open-source repos and tiered dataset fetch script with checksums.

Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01814GKVzZVL3FuF3DeiZXhv
 2026-09-27 15:20:06 +01:00