kemal
d163b34eca
- signal/normalization/numerics.py: vendor percentile, trimmean, MAD sigma (old/new), bisquare robust fits, float32/float64 typing and sequential sums reproduced exactly (ported from gtc2vcf, MIT) - signal/normalization/fit.py: outlier removal, offset, rotation, shear, scale, nn12/rr12, single-channel model for small Infinium I bins, exact and binned nearest-neighbour sweeps; vendor quirks marked QUIRK - signal/normalization/apply.py: affine transform and theta/R - pipeline/normalize.py + CLI `simurg normalize` - oracle GTCs rebuilt with -ffp-contract=off (aarch64 gcc fuses a*b+c by default, shifting the last bits); the old FMA oracle differed by <=4 ULP Result: transforms bit-identical to the oracle for all 36 GSA samples x 48 bins, for both normalization versions (3456/3456 bins, 0 ULP).
39 lines
2.0 KiB
Bash
Executable File
39 lines
2.0 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
# Build bcftools + gtc2vcf plugins (MIT) under refs/tools and generate oracle GTC files from the P0 GSA demo IDATs,
|
|
# once per normalization version (GenTrain 3 = normalization 1.2.0, GenTrain 2 = 1.1.2).
|
|
# The plugin is compiled with -ffp-contract=off: compilers may otherwise fuse a*b+c into FMA instructions (default
|
|
# on aarch64), which changes the last bits of the normalization transforms. Vendor .NET code and NumPy do not fuse.
|
|
# Usage: bash scripts/make_oracle_gtcs.sh [chip-folder ...] (default: every chip in the demo set)
|
|
set -euo pipefail
|
|
cd "$(dirname "$0")/.."
|
|
V=1.22
|
|
T=refs/tools
|
|
B=$T/bcftools-$V
|
|
if [[ ! -x $B/bcftools || ! -f $B/plugins/idat2gtc.so ]]; then
|
|
[[ -d refs/gtc2vcf ]] || bash scripts/bootstrap_refs.sh
|
|
mkdir -p $T
|
|
[[ -f $T/bcftools.tar.bz2 ]] || curl -sL -o $T/bcftools.tar.bz2 \
|
|
"https://github.com/samtools/bcftools/releases/download/$V/bcftools-$V.tar.bz2"
|
|
tar -xjf $T/bcftools.tar.bz2 -C $T
|
|
rm -f $B/plugins/{idat2gtc,gtc2vcf,affy2vcf}.{c,h}
|
|
cp refs/gtc2vcf/{idat2gtc,gtc2vcf,affy2vcf}.c refs/gtc2vcf/*.h $B/plugins/
|
|
(cd $B/htslib-$V && ./configure --disable-libcurl --disable-bz2 --disable-lzma --disable-ref-cache >/dev/null)
|
|
(cd $B && ./configure --without-gsl >/dev/null && make -j"$(nproc)" CFLAGS="-g -O2 -ffp-contract=off" >/dev/null)
|
|
fi
|
|
export BCFTOOLS_PLUGINS=$B/plugins
|
|
D=data/p0/gsa_v4_pgx
|
|
IDATS=$D/idats_Demo_36/idats_Demo_36
|
|
chips=("$@")
|
|
[[ ${#chips[@]} -eq 0 ]] && mapfile -t chips < <(find "$IDATS" -mindepth 1 -maxdepth 1 -type d | sort)
|
|
for v in 3 2; do
|
|
OUT=data/p0/oracle/gsa_gt${v}_nofma
|
|
mkdir -p "$OUT"
|
|
for chip in "${chips[@]}"; do
|
|
$B/bcftools +idat2gtc --gentrain-version $v \
|
|
--bpm $D/GSA-PGx-48v4-0_20079540_E2_bpm/GSA-PGx-48v4-0_20079540_E2.bpm \
|
|
--egt $D/GSA-PGx-48v4-0_20079540_E2_ClusterFile/GSA-PGx-48v4-0_20079540_E2_ClusterFile.egt \
|
|
--idats "$chip" --output "$OUT" 2>&1 | grep -E "^(Call rate|Gender|Writing)" || true
|
|
done
|
|
echo "oracle GTCs in $OUT: $(ls "$OUT"/*.gtc | wc -l)"
|
|
done
|