From 0b05b0d3462a051e0db08beb3a816c0f6831c356 Mon Sep 17 00:00:00 2001 From: Kemal Yaylali Date: Mon, 28 Sep 2026 23:53:59 +0100 Subject: [PATCH] P10 complete: packaging verified on fresh x86_64 and aarch64 VMs Wheels (pure + manylinux_2_28 kernels), sdist, Docker image and conda packages built and installed on fresh Ubuntu 24.04 machines; 225 unit tests pass against the installed package; the tutorial runs with pip (with and without kernels); GTC/VCF content digests are identical across pip, Docker and conda on both architectures and equal to the Pi's run. docs/validation/P10.md; guide: install from wheels, Docker, conda; exit code 141. --- CHANGELOG.md | 9 ++++++++- README.md | 3 ++- docs/user_guide.md | 28 ++++++++++++++++++++++++++- docs/validation/P10.md | 43 ++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 80 insertions(+), 3 deletions(-) create mode 100644 docs/validation/P10.md diff --git a/CHANGELOG.md b/CHANGELOG.md index 62de98d..e392aa5 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,6 +1,13 @@ # Changelog -## Unreleased (P9) +## Unreleased (P9, P10) + +**Packaging (P10).** `scripts/build_dist.sh` (sdist, pure wheel, manylinux_2_28 kernel wheels for x86_64 and +aarch64), `Dockerfile`, conda recipes (`packaging/conda/`). Verified on fresh x86_64 and aarch64 VMs: pip, Docker +and conda installs give identical outputs (`docs/validation/P10.md`). A closed stdout pipe now exits 141 silently +instead of reporting an internal error. + +**Performance (P9).** Optional compiled kernels `rust/simurg-core` (PyO3): fused GenCall calling and BAF/LRR, libm maps with CPython semantics, normalization nearest-neighbour searches, BPM parser. Byte-identical outputs to the pure-Python paths diff --git a/README.md b/README.md index 12c743f..a1795bf 100644 --- a/README.md +++ b/README.md @@ -23,7 +23,8 @@ differential-testing harness. Validation (`docs/validation/`): Commands: `simurg inspect`, `ingest`, `normalize`, `genotype`, `qc controls|samples|contamination|relatedness|loci|report|suggest-thresholds`, `gtc-to-vcf`, `export plink|bedgraph|finalreport|locus-summary`, `compare`. **User guide: `docs/user_guide.md`**; worked example: `examples/gsa_demo.sh`. P9 (performance) is done: optional Rust kernels, byte-identical outputs, -~80 GSA samples/min per server core (`docs/validation/P9.md`). Next (plan §18): P10 packaging, then v2 modules +~80 GSA samples/min per server core (`docs/validation/P9.md`). P10 (packaging) is done: wheels, Docker image and +conda recipes, verified on fresh x86_64 and aarch64 machines (`docs/validation/P10.md`). Next (plan §18): v2 modules (cluster training, CN/LOH, PGx). ## Quick start diff --git a/docs/user_guide.md b/docs/user_guide.md index ef76d02..3d41c4a 100644 --- a/docs/user_guide.md +++ b/docs/user_guide.md @@ -28,6 +28,31 @@ uv pip install ./rust/simurg-core ``` Without them the same code runs in pure Python/NumPy; `SIMURG_PURE_PYTHON=1` forces that path. + +**From release files** (`bash scripts/build_dist.sh` writes them to `dist/`: a pure wheel, an sdist and +`manylinux_2_28` kernel wheels for x86_64 and aarch64, glibc ≥ 2.28): + +```bash +python3 -m venv simurg-env && simurg-env/bin/pip install simurg-1.0.0-py3-none-any.whl \ + simurg_core-1.0.0-cp312-abi3-manylinux_2_28_$(uname -m).whl # the second wheel is optional +``` + +**Docker** (image ≈ 550 MB, kernels included, runs as a non-root user; data stay in the mounted folder): + +```bash +docker build -t simurg:1.0.0 . +docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/work simurg:1.0.0 \ + genotype --bpm M.bpm --egt C.egt --idats idats --out run/gtc +``` + +**conda** (recipes in `packaging/conda/`): + +```bash +conda build -c conda-forge --output-folder conda-bld packaging/conda/simurg-core packaging/conda/simurg +conda create -n simurg -c ./conda-bld -c conda-forge simurg simurg-core +``` + +All four installs give identical results on x86_64 and aarch64 (`docs/validation/P10.md`). Only pure-Python dependencies are needed (numpy, pyarrow, typer, rich, pyyaml, openpyxl). VCF/BCF, tabix/CSI indexes and PLINK files are written by SimurgArray itself; bcftools/plink2 are not required (they are used only as test oracles). @@ -167,5 +192,6 @@ A nightly differential run (`scripts/nightly.py`) re-checks these against the or Errors and warnings carry a stable code (`E_…`, `W_…`) — all are listed in `docs/messages.md` with their spec item. Add `--debug` for a traceback or `--log-json` for machine-readable logs. Exit codes (CLI-014): 0 success, 1 internal error, 2 usage or pre-flight validation error (nothing processed), 3 some samples failed (or -`compare --fail-on-discrepancy` found discrepancies), 4 no sample succeeded, 130/143 interrupted. Findings about vendor behaviour that SimurgArray reproduces +`compare --fail-on-discrepancy` found discrepancies), 4 no sample succeeded, 130/143 interrupted, 141 the reader of +stdout went away (e.g. `simurg inspect … | head`; nothing is reported). Findings about vendor behaviour that SimurgArray reproduces (or deliberately does not) are in `docs/known_unknowns.md`. diff --git a/docs/validation/P10.md b/docs/validation/P10.md new file mode 100644 index 0000000..80ed61b --- /dev/null +++ b/docs/validation/P10.md @@ -0,0 +1,43 @@ +# Phase P10 validation — packaging and fresh-machine acceptance + +Date: 2026-09-28. Artifacts built and tested on fresh AWS Ubuntu 24.04 VMs (4 vCPU): aarch64 c7g.xlarge and x86_64 +m6a.xlarge / c6a.xlarge (one x86 spot instance was reclaimed by AWS after 4 min and relaunched). Cloud cost ≈ $0.10. + +## Artifacts + +| Artifact | How | Notes | +|---|---|---| +| `simurg-1.0.0-py3-none-any.whl`, `simurg-1.0.0.tar.gz` | `uv build` (hatchling) | QC report assets, profiles, `LICENSE` and `THIRD_PARTY_NOTICES.md` inside | +| `simurg_core-1.0.0-cp312-abi3-manylinux_2_28_{aarch64,x86_64}.whl` | `maturin --zig --compatibility manylinux_2_28` (both built on either machine) | one wheel per architecture for every Python ≥ 3.12; glibc ≥ 2.28 | +| Docker image (`Dockerfile`) | multi-stage, `python:3.12-slim-bookworm`, non-root user | 545 MB (x86_64), 551 MB (aarch64) | +| conda packages (`packaging/conda/`) | `conda build -c conda-forge` | `simurg` (noarch) + `simurg-core` (Rust compiler from conda-forge) | + +`scripts/build_dist.sh` builds the wheels and sdist and writes `dist/SHA256SUMS`. A wheel built on the Pi is not +portable (glibc 2.41 symbols such as `exp@GLIBC_2.29`), hence the zig-linked manylinux build. + +## Acceptance (plan §18 P10: clean install + tutorial works on a fresh VM) + +`scripts/cloud/tasks/p10.sh` (and `p10t.sh`, the same without Docker/conda) on each architecture: + +| Step | aarch64 | x86_64 | +|---|---|---| +| Build sdist, pure wheel, both kernel wheels | ok | ok | +| `pip install` into a new venv of the system Python 3.12.3 (with / without kernels) | ok / ok | ok / ok | +| Unit tests against the installed package (225) | pass | pass | +| Tutorial `examples/gsa_demo.sh` on 6 demo samples, pip install with kernels | ok (506 s) | ok (521 s) | +| Tutorial, pure-Python install | ok (539 s) | ok (494 s) | +| Docker build + `genotype` + `gtc-to-vcf` | ok | ok | +| conda build + install + `genotype` | ok | ok | + +Content digests (`scripts/p10/digest.py`: every numeric GTC field and sample statistic with NaNs canonicalised, all +VCF data lines) of the 6 GTCs and 6 VCFs are **identical across all of them**: pip with kernels, pip pure, Docker and +conda, on aarch64 and on x86_64, and equal to the Raspberry Pi's own run of the same tutorial. The x86_64 results come +from AMD EPYC (m6a/c6a) machines; bit-exactness had so far only been shown on ARM. + +## Found and fixed + +* `simurg inspect … | head` made the CLI report `E_INTERNAL: BrokenPipeError` and exit 1, which stopped the tutorial + under `set -o pipefail` (it passed on the Pi only by timing luck). A closed stdout now exits with 141 (like a tool + killed by SIGPIPE) and prints nothing; the tutorial reads with `sed` instead of `head`; unit test added. +* The `fast` extra (`simurg[fast]` → `simurg-core`) was dropped: `uv` must resolve every extra, and `simurg-core` is + not on a package index. The kernels are installed as their own wheel.