P10 complete: packaging verified on fresh x86_64 and aarch64 VMs
Wheels (pure + manylinux_2_28 kernels), sdist, Docker image and conda packages built and installed on fresh Ubuntu 24.04 machines; 225 unit tests pass against the installed package; the tutorial runs with pip (with and without kernels); GTC/VCF content digests are identical across pip, Docker and conda on both architectures and equal to the Pi's run. docs/validation/P10.md; guide: install from wheels, Docker, conda; exit code 141.
This commit is contained in:
1 parent
522429b731
commit
0b05b0d346
4 files changed
+80
-3
No files matched your search
+8
-1
@@ -1,6 +1,13 @@
|
|||||||
# Changelog
|
# Changelog
|
||||||
|
|
||||||
## Unreleased (P9)
|
## Unreleased (P9, P10)
|
||||||
|
|
||||||
|
**Packaging (P10).** `scripts/build_dist.sh` (sdist, pure wheel, manylinux_2_28 kernel wheels for x86_64 and
|
||||||
|
aarch64), `Dockerfile`, conda recipes (`packaging/conda/`). Verified on fresh x86_64 and aarch64 VMs: pip, Docker
|
||||||
|
and conda installs give identical outputs (`docs/validation/P10.md`). A closed stdout pipe now exits 141 silently
|
||||||
|
instead of reporting an internal error.
|
||||||
|
|
||||||
|
**Performance (P9).**
|
||||||
|
|
||||||
Optional compiled kernels `rust/simurg-core` (PyO3): fused GenCall calling and BAF/LRR, libm maps with CPython
|
Optional compiled kernels `rust/simurg-core` (PyO3): fused GenCall calling and BAF/LRR, libm maps with CPython
|
||||||
semantics, normalization nearest-neighbour searches, BPM parser. Byte-identical outputs to the pure-Python paths
|
semantics, normalization nearest-neighbour searches, BPM parser. Byte-identical outputs to the pure-Python paths
|
||||||
|
|||||||
@@ -23,7 +23,8 @@ differential-testing harness. Validation (`docs/validation/`):
|
|||||||
Commands: `simurg inspect`, `ingest`, `normalize`, `genotype`, `qc controls|samples|contamination|relatedness|loci|report|suggest-thresholds`,
|
Commands: `simurg inspect`, `ingest`, `normalize`, `genotype`, `qc controls|samples|contamination|relatedness|loci|report|suggest-thresholds`,
|
||||||
`gtc-to-vcf`, `export plink|bedgraph|finalreport|locus-summary`, `compare`. **User guide: `docs/user_guide.md`**;
|
`gtc-to-vcf`, `export plink|bedgraph|finalreport|locus-summary`, `compare`. **User guide: `docs/user_guide.md`**;
|
||||||
worked example: `examples/gsa_demo.sh`. P9 (performance) is done: optional Rust kernels, byte-identical outputs,
|
worked example: `examples/gsa_demo.sh`. P9 (performance) is done: optional Rust kernels, byte-identical outputs,
|
||||||
~80 GSA samples/min per server core (`docs/validation/P9.md`). Next (plan §18): P10 packaging, then v2 modules
|
~80 GSA samples/min per server core (`docs/validation/P9.md`). P10 (packaging) is done: wheels, Docker image and
|
||||||
|
conda recipes, verified on fresh x86_64 and aarch64 machines (`docs/validation/P10.md`). Next (plan §18): v2 modules
|
||||||
(cluster training, CN/LOH, PGx).
|
(cluster training, CN/LOH, PGx).
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|||||||
+27
-1
@@ -28,6 +28,31 @@ uv pip install ./rust/simurg-core
|
|||||||
```
|
```
|
||||||
|
|
||||||
Without them the same code runs in pure Python/NumPy; `SIMURG_PURE_PYTHON=1` forces that path.
|
Without them the same code runs in pure Python/NumPy; `SIMURG_PURE_PYTHON=1` forces that path.
|
||||||
|
|
||||||
|
**From release files** (`bash scripts/build_dist.sh` writes them to `dist/`: a pure wheel, an sdist and
|
||||||
|
`manylinux_2_28` kernel wheels for x86_64 and aarch64, glibc ≥ 2.28):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv simurg-env && simurg-env/bin/pip install simurg-1.0.0-py3-none-any.whl \
|
||||||
|
simurg_core-1.0.0-cp312-abi3-manylinux_2_28_$(uname -m).whl # the second wheel is optional
|
||||||
|
```
|
||||||
|
|
||||||
|
**Docker** (image ≈ 550 MB, kernels included, runs as a non-root user; data stay in the mounted folder):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker build -t simurg:1.0.0 .
|
||||||
|
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/work simurg:1.0.0 \
|
||||||
|
genotype --bpm M.bpm --egt C.egt --idats idats --out run/gtc
|
||||||
|
```
|
||||||
|
|
||||||
|
**conda** (recipes in `packaging/conda/`):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
conda build -c conda-forge --output-folder conda-bld packaging/conda/simurg-core packaging/conda/simurg
|
||||||
|
conda create -n simurg -c ./conda-bld -c conda-forge simurg simurg-core
|
||||||
|
```
|
||||||
|
|
||||||
|
All four installs give identical results on x86_64 and aarch64 (`docs/validation/P10.md`).
|
||||||
Only pure-Python dependencies are needed (numpy, pyarrow, typer, rich, pyyaml, openpyxl). VCF/BCF, tabix/CSI
|
Only pure-Python dependencies are needed (numpy, pyarrow, typer, rich, pyyaml, openpyxl). VCF/BCF, tabix/CSI
|
||||||
indexes and PLINK files are written by SimurgArray itself; bcftools/plink2 are not required (they are used only
|
indexes and PLINK files are written by SimurgArray itself; bcftools/plink2 are not required (they are used only
|
||||||
as test oracles).
|
as test oracles).
|
||||||
@@ -167,5 +192,6 @@ A nightly differential run (`scripts/nightly.py`) re-checks these against the or
|
|||||||
Errors and warnings carry a stable code (`E_…`, `W_…`) — all are listed in `docs/messages.md` with their spec item.
|
Errors and warnings carry a stable code (`E_…`, `W_…`) — all are listed in `docs/messages.md` with their spec item.
|
||||||
Add `--debug` for a traceback or `--log-json` for machine-readable logs. Exit codes (CLI-014): 0 success,
|
Add `--debug` for a traceback or `--log-json` for machine-readable logs. Exit codes (CLI-014): 0 success,
|
||||||
1 internal error, 2 usage or pre-flight validation error (nothing processed), 3 some samples failed (or
|
1 internal error, 2 usage or pre-flight validation error (nothing processed), 3 some samples failed (or
|
||||||
`compare --fail-on-discrepancy` found discrepancies), 4 no sample succeeded, 130/143 interrupted. Findings about vendor behaviour that SimurgArray reproduces
|
`compare --fail-on-discrepancy` found discrepancies), 4 no sample succeeded, 130/143 interrupted, 141 the reader of
|
||||||
|
stdout went away (e.g. `simurg inspect … | head`; nothing is reported). Findings about vendor behaviour that SimurgArray reproduces
|
||||||
(or deliberately does not) are in `docs/known_unknowns.md`.
|
(or deliberately does not) are in `docs/known_unknowns.md`.
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
# Phase P10 validation — packaging and fresh-machine acceptance
|
||||||
|
|
||||||
|
Date: 2026-09-28. Artifacts built and tested on fresh AWS Ubuntu 24.04 VMs (4 vCPU): aarch64 c7g.xlarge and x86_64
|
||||||
|
m6a.xlarge / c6a.xlarge (one x86 spot instance was reclaimed by AWS after 4 min and relaunched). Cloud cost ≈ $0.10.
|
||||||
|
|
||||||
|
## Artifacts
|
||||||
|
|
||||||
|
| Artifact | How | Notes |
|
||||||
|
|---|---|---|
|
||||||
|
| `simurg-1.0.0-py3-none-any.whl`, `simurg-1.0.0.tar.gz` | `uv build` (hatchling) | QC report assets, profiles, `LICENSE` and `THIRD_PARTY_NOTICES.md` inside |
|
||||||
|
| `simurg_core-1.0.0-cp312-abi3-manylinux_2_28_{aarch64,x86_64}.whl` | `maturin --zig --compatibility manylinux_2_28` (both built on either machine) | one wheel per architecture for every Python ≥ 3.12; glibc ≥ 2.28 |
|
||||||
|
| Docker image (`Dockerfile`) | multi-stage, `python:3.12-slim-bookworm`, non-root user | 545 MB (x86_64), 551 MB (aarch64) |
|
||||||
|
| conda packages (`packaging/conda/`) | `conda build -c conda-forge` | `simurg` (noarch) + `simurg-core` (Rust compiler from conda-forge) |
|
||||||
|
|
||||||
|
`scripts/build_dist.sh` builds the wheels and sdist and writes `dist/SHA256SUMS`. A wheel built on the Pi is not
|
||||||
|
portable (glibc 2.41 symbols such as `exp@GLIBC_2.29`), hence the zig-linked manylinux build.
|
||||||
|
|
||||||
|
## Acceptance (plan §18 P10: clean install + tutorial works on a fresh VM)
|
||||||
|
|
||||||
|
`scripts/cloud/tasks/p10.sh` (and `p10t.sh`, the same without Docker/conda) on each architecture:
|
||||||
|
|
||||||
|
| Step | aarch64 | x86_64 |
|
||||||
|
|---|---|---|
|
||||||
|
| Build sdist, pure wheel, both kernel wheels | ok | ok |
|
||||||
|
| `pip install` into a new venv of the system Python 3.12.3 (with / without kernels) | ok / ok | ok / ok |
|
||||||
|
| Unit tests against the installed package (225) | pass | pass |
|
||||||
|
| Tutorial `examples/gsa_demo.sh` on 6 demo samples, pip install with kernels | ok (506 s) | ok (521 s) |
|
||||||
|
| Tutorial, pure-Python install | ok (539 s) | ok (494 s) |
|
||||||
|
| Docker build + `genotype` + `gtc-to-vcf` | ok | ok |
|
||||||
|
| conda build + install + `genotype` | ok | ok |
|
||||||
|
|
||||||
|
Content digests (`scripts/p10/digest.py`: every numeric GTC field and sample statistic with NaNs canonicalised, all
|
||||||
|
VCF data lines) of the 6 GTCs and 6 VCFs are **identical across all of them**: pip with kernels, pip pure, Docker and
|
||||||
|
conda, on aarch64 and on x86_64, and equal to the Raspberry Pi's own run of the same tutorial. The x86_64 results come
|
||||||
|
from AMD EPYC (m6a/c6a) machines; bit-exactness had so far only been shown on ARM.
|
||||||
|
|
||||||
|
## Found and fixed
|
||||||
|
|
||||||
|
* `simurg inspect … | head` made the CLI report `E_INTERNAL: BrokenPipeError` and exit 1, which stopped the tutorial
|
||||||
|
under `set -o pipefail` (it passed on the Pi only by timing luck). A closed stdout now exits with 141 (like a tool
|
||||||
|
killed by SIGPIPE) and prints nothing; the tutorial reads with `sed` instead of `head`; unit test added.
|
||||||
|
* The `fast` extra (`simurg[fast]` → `simurg-core`) was dropped: `uv` must resolve every extra, and `simurg-core` is
|
||||||
|
not on a package index. The kernels are installed as their own wheel.
|
||||||
Reference in new issue
Block a user