P10 complete: packaging verified on fresh x86_64 and aarch64 VMs
Wheels (pure + manylinux_2_28 kernels), sdist, Docker image and conda packages built and installed on fresh Ubuntu 24.04 machines; 225 unit tests pass against the installed package; the tutorial runs with pip (with and without kernels); GTC/VCF content digests are identical across pip, Docker and conda on both architectures and equal to the Pi's run. docs/validation/P10.md; guide: install from wheels, Docker, conda; exit code 141.
This commit is contained in:
1 parent
cbcb99f8d2
commit
b11c3706ac
4 files changed
+80
-3
No files matched your search
+27
-1
@@ -28,6 +28,31 @@ uv pip install ./rust/simurg-core
|
||||
```
|
||||
|
||||
Without them the same code runs in pure Python/NumPy; `SIMURG_PURE_PYTHON=1` forces that path.
|
||||
|
||||
**From release files** (`bash scripts/build_dist.sh` writes them to `dist/`: a pure wheel, an sdist and
|
||||
`manylinux_2_28` kernel wheels for x86_64 and aarch64, glibc ≥ 2.28):
|
||||
|
||||
```bash
|
||||
python3 -m venv simurg-env && simurg-env/bin/pip install simurg-1.0.0-py3-none-any.whl \
|
||||
simurg_core-1.0.0-cp312-abi3-manylinux_2_28_$(uname -m).whl # the second wheel is optional
|
||||
```
|
||||
|
||||
**Docker** (image ≈ 550 MB, kernels included, runs as a non-root user; data stay in the mounted folder):
|
||||
|
||||
```bash
|
||||
docker build -t simurg:1.0.0 .
|
||||
docker run --rm --user "$(id -u):$(id -g)" -v "$PWD":/work simurg:1.0.0 \
|
||||
genotype --bpm M.bpm --egt C.egt --idats idats --out run/gtc
|
||||
```
|
||||
|
||||
**conda** (recipes in `packaging/conda/`):
|
||||
|
||||
```bash
|
||||
conda build -c conda-forge --output-folder conda-bld packaging/conda/simurg-core packaging/conda/simurg
|
||||
conda create -n simurg -c ./conda-bld -c conda-forge simurg simurg-core
|
||||
```
|
||||
|
||||
All four installs give identical results on x86_64 and aarch64 (`docs/validation/P10.md`).
|
||||
Only pure-Python dependencies are needed (numpy, pyarrow, typer, rich, pyyaml, openpyxl). VCF/BCF, tabix/CSI
|
||||
indexes and PLINK files are written by SimurgArray itself; bcftools/plink2 are not required (they are used only
|
||||
as test oracles).
|
||||
@@ -167,5 +192,6 @@ A nightly differential run (`scripts/nightly.py`) re-checks these against the or
|
||||
Errors and warnings carry a stable code (`E_…`, `W_…`) — all are listed in `docs/messages.md` with their spec item.
|
||||
Add `--debug` for a traceback or `--log-json` for machine-readable logs. Exit codes (CLI-014): 0 success,
|
||||
1 internal error, 2 usage or pre-flight validation error (nothing processed), 3 some samples failed (or
|
||||
`compare --fail-on-discrepancy` found discrepancies), 4 no sample succeeded, 130/143 interrupted. Findings about vendor behaviour that SimurgArray reproduces
|
||||
`compare --fail-on-discrepancy` found discrepancies), 4 no sample succeeded, 130/143 interrupted, 141 the reader of
|
||||
stdout went away (e.g. `simurg inspect … | head`; nothing is reported). Findings about vendor behaviour that SimurgArray reproduces
|
||||
(or deliberately does not) are in `docs/known_unknowns.md`.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Phase P10 validation — packaging and fresh-machine acceptance
|
||||
|
||||
Date: 2026-09-28. Artifacts built and tested on fresh AWS Ubuntu 24.04 VMs (4 vCPU): aarch64 c7g.xlarge and x86_64
|
||||
m6a.xlarge / c6a.xlarge (one x86 spot instance was reclaimed by AWS after 4 min and relaunched). Cloud cost ≈ $0.10.
|
||||
|
||||
## Artifacts
|
||||
|
||||
| Artifact | How | Notes |
|
||||
|---|---|---|
|
||||
| `simurg-1.0.0-py3-none-any.whl`, `simurg-1.0.0.tar.gz` | `uv build` (hatchling) | QC report assets, profiles, `LICENSE` and `THIRD_PARTY_NOTICES.md` inside |
|
||||
| `simurg_core-1.0.0-cp312-abi3-manylinux_2_28_{aarch64,x86_64}.whl` | `maturin --zig --compatibility manylinux_2_28` (both built on either machine) | one wheel per architecture for every Python ≥ 3.12; glibc ≥ 2.28 |
|
||||
| Docker image (`Dockerfile`) | multi-stage, `python:3.12-slim-bookworm`, non-root user | 545 MB (x86_64), 551 MB (aarch64) |
|
||||
| conda packages (`packaging/conda/`) | `conda build -c conda-forge` | `simurg` (noarch) + `simurg-core` (Rust compiler from conda-forge) |
|
||||
|
||||
`scripts/build_dist.sh` builds the wheels and sdist and writes `dist/SHA256SUMS`. A wheel built on the Pi is not
|
||||
portable (glibc 2.41 symbols such as `exp@GLIBC_2.29`), hence the zig-linked manylinux build.
|
||||
|
||||
## Acceptance (plan §18 P10: clean install + tutorial works on a fresh VM)
|
||||
|
||||
`scripts/cloud/tasks/p10.sh` (and `p10t.sh`, the same without Docker/conda) on each architecture:
|
||||
|
||||
| Step | aarch64 | x86_64 |
|
||||
|---|---|---|
|
||||
| Build sdist, pure wheel, both kernel wheels | ok | ok |
|
||||
| `pip install` into a new venv of the system Python 3.12.3 (with / without kernels) | ok / ok | ok / ok |
|
||||
| Unit tests against the installed package (225) | pass | pass |
|
||||
| Tutorial `examples/gsa_demo.sh` on 6 demo samples, pip install with kernels | ok (506 s) | ok (521 s) |
|
||||
| Tutorial, pure-Python install | ok (539 s) | ok (494 s) |
|
||||
| Docker build + `genotype` + `gtc-to-vcf` | ok | ok |
|
||||
| conda build + install + `genotype` | ok | ok |
|
||||
|
||||
Content digests (`scripts/p10/digest.py`: every numeric GTC field and sample statistic with NaNs canonicalised, all
|
||||
VCF data lines) of the 6 GTCs and 6 VCFs are **identical across all of them**: pip with kernels, pip pure, Docker and
|
||||
conda, on aarch64 and on x86_64, and equal to the Raspberry Pi's own run of the same tutorial. The x86_64 results come
|
||||
from AMD EPYC (m6a/c6a) machines; bit-exactness had so far only been shown on ARM.
|
||||
|
||||
## Found and fixed
|
||||
|
||||
* `simurg inspect … | head` made the CLI report `E_INTERNAL: BrokenPipeError` and exit 1, which stopped the tutorial
|
||||
under `set -o pipefail` (it passed on the Pi only by timing luck). A closed stdout now exits with 141 (like a tool
|
||||
killed by SIGPIPE) and prints nothing; the tutorial reads with `sed` instead of `head`; unit test added.
|
||||
* The `fast` extra (`simurg[fast]` → `simurg-core`) was dropped: `uv` must resolve every extra, and `simurg-core` is
|
||||
not on a package index. The kernels are installed as their own wheel.
|
||||
Reference in new issue
Block a user