P3: normalization fit (1.2.0 and 1.1.2), apply, polar, simurg normalize
- signal/normalization/numerics.py: vendor percentile, trimmean, MAD sigma (old/new), bisquare robust fits, float32/float64 typing and sequential sums reproduced exactly (ported from gtc2vcf, MIT) - signal/normalization/fit.py: outlier removal, offset, rotation, shear, scale, nn12/rr12, single-channel model for small Infinium I bins, exact and binned nearest-neighbour sweeps; vendor quirks marked QUIRK - signal/normalization/apply.py: affine transform and theta/R - pipeline/normalize.py + CLI `simurg normalize` - oracle GTCs rebuilt with -ffp-contract=off (aarch64 gcc fuses a*b+c by default, shifting the last bits); the old FMA oracle differed by <=4 ULP Result: transforms bit-identical to the oracle for all 36 GSA samples x 48 bins, for both normalization versions (3456/3456 bins, 0 ULP).
This commit is contained in:
1 parent
f1e22ab339
commit
95e4e41560
12 files changed
+1097
-14
No files matched your search
+15
-11
@@ -1,5 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Build bcftools + gtc2vcf plugins (MIT) under refs/tools and generate oracle GTC files from the P0 GSA demo IDATs.
|
||||
# Build bcftools + gtc2vcf plugins (MIT) under refs/tools and generate oracle GTC files from the P0 GSA demo IDATs,
|
||||
# once per normalization version (GenTrain 3 = normalization 1.2.0, GenTrain 2 = 1.1.2).
|
||||
# The plugin is compiled with -ffp-contract=off: compilers may otherwise fuse a*b+c into FMA instructions (default
|
||||
# on aarch64), which changes the last bits of the normalization transforms. Vendor .NET code and NumPy do not fuse.
|
||||
# Usage: bash scripts/make_oracle_gtcs.sh [chip-folder ...] (default: every chip in the demo set)
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
@@ -15,20 +18,21 @@ if [[ ! -x $B/bcftools || ! -f $B/plugins/idat2gtc.so ]]; then
|
||||
rm -f $B/plugins/{idat2gtc,gtc2vcf,affy2vcf}.{c,h}
|
||||
cp refs/gtc2vcf/{idat2gtc,gtc2vcf,affy2vcf}.c refs/gtc2vcf/*.h $B/plugins/
|
||||
(cd $B/htslib-$V && ./configure --disable-libcurl --disable-bz2 --disable-lzma --disable-ref-cache >/dev/null)
|
||||
(cd $B && ./configure --without-gsl >/dev/null && make -j"$(nproc)" >/dev/null)
|
||||
(cd $B && ./configure --without-gsl >/dev/null && make -j"$(nproc)" CFLAGS="-g -O2 -ffp-contract=off" >/dev/null)
|
||||
fi
|
||||
export BCFTOOLS_PLUGINS=$B/plugins
|
||||
D=data/p0/gsa_v4_pgx
|
||||
IDATS=$D/idats_Demo_36/idats_Demo_36
|
||||
OUT=data/p0/oracle/gsa_gtc_gtc2vcf
|
||||
mkdir -p "$OUT"
|
||||
chips=("$@")
|
||||
[[ ${#chips[@]} -eq 0 ]] && mapfile -t chips < <(find "$IDATS" -mindepth 1 -maxdepth 1 -type d | sort)
|
||||
for chip in "${chips[@]}"; do
|
||||
$B/bcftools +idat2gtc \
|
||||
--bpm $D/GSA-PGx-48v4-0_20079540_E2_bpm/GSA-PGx-48v4-0_20079540_E2.bpm \
|
||||
--egt $D/GSA-PGx-48v4-0_20079540_E2_ClusterFile/GSA-PGx-48v4-0_20079540_E2_ClusterFile.egt \
|
||||
--idats "$chip" --output "$OUT" 2>&1 | grep -E "^(Call rate|Gender|Writing)" || true
|
||||
for v in 3 2; do
|
||||
OUT=data/p0/oracle/gsa_gt${v}_nofma
|
||||
mkdir -p "$OUT"
|
||||
for chip in "${chips[@]}"; do
|
||||
$B/bcftools +idat2gtc --gentrain-version $v \
|
||||
--bpm $D/GSA-PGx-48v4-0_20079540_E2_bpm/GSA-PGx-48v4-0_20079540_E2.bpm \
|
||||
--egt $D/GSA-PGx-48v4-0_20079540_E2_ClusterFile/GSA-PGx-48v4-0_20079540_E2_ClusterFile.egt \
|
||||
--idats "$chip" --output "$OUT" 2>&1 | grep -E "^(Call rate|Gender|Writing)" || true
|
||||
done
|
||||
echo "oracle GTCs in $OUT: $(ls "$OUT"/*.gtc | wc -l)"
|
||||
done
|
||||
$B/bcftools +idat2gtc --version 2>&1 | head -2 || true
|
||||
echo "oracle GTCs in $OUT: $(ls "$OUT"/*.gtc | wc -l)"
|
||||
Reference in new issue
Block a user