P3: normalization fit (1.2.0 and 1.1.2), apply, polar, simurg normalize

- signal/normalization/numerics.py: vendor percentile, trimmean, MAD sigma
  (old/new), bisquare robust fits, float32/float64 typing and sequential
  sums reproduced exactly (ported from gtc2vcf, MIT)
- signal/normalization/fit.py: outlier removal, offset, rotation, shear,
  scale, nn12/rr12, single-channel model for small Infinium I bins,
  exact and binned nearest-neighbour sweeps; vendor quirks marked QUIRK
- signal/normalization/apply.py: affine transform and theta/R
- pipeline/normalize.py + CLI `simurg normalize`
- oracle GTCs rebuilt with -ffp-contract=off (aarch64 gcc fuses a*b+c by
  default, shifting the last bits); the old FMA oracle differed by <=4 ULP

Result: transforms bit-identical to the oracle for all 36 GSA samples x
48 bins, for both normalization versions (3456/3456 bins, 0 ULP).
This commit is contained in:
kemal committed 2026-09-27 22:51:06 +01:00
1 parent 85229fba84
commit d163b34eca
12 files changed
+1097 -14

No files matched your search

+15 -11
View File
@@ -1,5 +1,8 @@
#!/usr/bin/env bash
# Build bcftools + gtc2vcf plugins (MIT) under refs/tools and generate oracle GTC files from the P0 GSA demo IDATs.
# Build bcftools + gtc2vcf plugins (MIT) under refs/tools and generate oracle GTC files from the P0 GSA demo IDATs,
# once per normalization version (GenTrain 3 = normalization 1.2.0, GenTrain 2 = 1.1.2).
# The plugin is compiled with -ffp-contract=off: compilers may otherwise fuse a*b+c into FMA instructions (default
# on aarch64), which changes the last bits of the normalization transforms. Vendor .NET code and NumPy do not fuse.
# Usage: bash scripts/make_oracle_gtcs.sh [chip-folder ...] (default: every chip in the demo set)
set -euo pipefail
cd "$(dirname "$0")/.."
@@ -15,20 +18,21 @@ if [[ ! -x $B/bcftools || ! -f $B/plugins/idat2gtc.so ]]; then
rm -f $B/plugins/{idat2gtc,gtc2vcf,affy2vcf}.{c,h}
cp refs/gtc2vcf/{idat2gtc,gtc2vcf,affy2vcf}.c refs/gtc2vcf/*.h $B/plugins/
(cd $B/htslib-$V && ./configure --disable-libcurl --disable-bz2 --disable-lzma --disable-ref-cache >/dev/null)
(cd $B && ./configure --without-gsl >/dev/null && make -j"$(nproc)" >/dev/null)
(cd $B && ./configure --without-gsl >/dev/null && make -j"$(nproc)" CFLAGS="-g -O2 -ffp-contract=off" >/dev/null)
fi
export BCFTOOLS_PLUGINS=$B/plugins
D=data/p0/gsa_v4_pgx
IDATS=$D/idats_Demo_36/idats_Demo_36
OUT=data/p0/oracle/gsa_gtc_gtc2vcf
mkdir -p "$OUT"
chips=("$@")
[[ ${#chips[@]} -eq 0 ]] && mapfile -t chips < <(find "$IDATS" -mindepth 1 -maxdepth 1 -type d | sort)
for chip in "${chips[@]}"; do
$B/bcftools +idat2gtc \
--bpm $D/GSA-PGx-48v4-0_20079540_E2_bpm/GSA-PGx-48v4-0_20079540_E2.bpm \
--egt $D/GSA-PGx-48v4-0_20079540_E2_ClusterFile/GSA-PGx-48v4-0_20079540_E2_ClusterFile.egt \
--idats "$chip" --output "$OUT" 2>&1 | grep -E "^(Call rate|Gender|Writing)" || true
for v in 3 2; do
OUT=data/p0/oracle/gsa_gt${v}_nofma
mkdir -p "$OUT"
for chip in "${chips[@]}"; do
$B/bcftools +idat2gtc --gentrain-version $v \
--bpm $D/GSA-PGx-48v4-0_20079540_E2_bpm/GSA-PGx-48v4-0_20079540_E2.bpm \
--egt $D/GSA-PGx-48v4-0_20079540_E2_ClusterFile/GSA-PGx-48v4-0_20079540_E2_ClusterFile.egt \
--idats "$chip" --output "$OUT" 2>&1 | grep -E "^(Call rate|Gender|Writing)" || true
done
echo "oracle GTCs in $OUT: $(ls "$OUT"/*.gtc | wc -l)"
done
$B/bcftools +idat2gtc --version 2>&1 | head -2 || true
echo "oracle GTCs in $OUT: $(ls "$OUT"/*.gtc | wc -l)"