Kemal Yaylali
e76ae847a1
A review of the ranking's arithmetic found four things wrong, all of which made the score look better informed than it was. Measurements below are from this repo, not estimates. **Components now abstain instead of inventing a number.** A run without a VEP cache returns no allele frequencies, and rarity_score(None) read that as "absent from gnomAD, therefore maximally rare" and awarded every variant a free 0.25. jobs.has_frequencies / has_effect_scores record what the run actually produced, absent components are dropped from the weighted mean, and the remaining weights are renormalised so the score keeps its meaning. The UI shows "not looked up" rather than a bar, and the funnel stops calling a step "rare" when nothing was filtered. **Allele frequency is no longer a model feature.** It dominated: the same missense variant scored 0.887 at AF 0 and 0.0003 at AF 0.01. That double- counted, because the ranking already scores frequency explicitly, putting ~45% of every rank on one measurement; and it was circular, because ACMG assigns ClinVar's benign labels using frequency (BA1/BS1). Retraining without it moves missense AUROC from 0.872 to 0.500 — exactly random. The old figure was allele frequency, not variant-effect knowledge. The model therefore abstains unless CADD or AlphaMissense is present, since otherwise it only restates the consequence class. **Phenotype matching is weighted by information content** and HPO annotations are propagated up the ontology. Counting terms alike let "global developmental delay" (IC 0.93) count as much as "dilated left subclavian artery" (IC 7.88). **A real bug in the propagation, found by checking it.** The ancestor walk read a pre-order DFS backwards, which on a DAG lets a term resolve before one of its parents and inherit that parent alone instead of its lineage. It dropped 399 terms out of the phenotype branch, Camptodactyly and Chiari malformation among them. Now a true post-order, tested against a reference transitive closure. The ontology arithmetic moved to rarelens_ml.hpo so it is covered by tests, and rarelens_ml.benchmark measures the whole thing: across 10,178 published cases the causal gene ranks first 45.9-81.0% of the time against 5,269 genes, versus 0.02% for chance. docs/data.md reports that with its contamination (HPO's annotations come from these same case reports), and includes the measurement showing information-content weighting earns its place while propagation does not - kept anyway, for a reason the docs argue rather than assume.
108 lines
5.1 KiB
Makefile
108 lines
5.1 KiB
Makefile
.PHONY: up down clean migrate test lint data hpo demo-case published-case benchmark training-set train loader pipeline annotate images kind serverless-deploy serverless-destroy gcp-configure gcp-secrets
|
|
|
|
VCF ?= data/example.vcf.gz
|
|
MLFLOW_URI ?= http://localhost:5001
|
|
DB_CONTAINER ?= rarelens-db-1
|
|
BENCH_DIR ?= data
|
|
TAG ?= latest
|
|
# The loader container reaches docker-compose's Postgres through the host.
|
|
HOST_DB_URL ?= postgresql://rarelens:[email protected]:5432/rarelens
|
|
|
|
up:
|
|
docker compose up -d --build
|
|
|
|
down:
|
|
docker compose down
|
|
|
|
clean: ## also deletes the Postgres and MLflow volumes
|
|
docker compose down -v
|
|
|
|
migrate:
|
|
docker compose exec api alembic upgrade head
|
|
|
|
test:
|
|
cd api && uv run --extra dev pytest -q
|
|
cd ml && uv run --extra dev pytest -q
|
|
cd pipeline && uv run --no-project --with-requirements requirements.txt --with pytest --with pgserver pytest -q tests
|
|
cd web && npm test
|
|
|
|
lint:
|
|
cd api && uv run --extra dev ruff check . && uv run --extra dev mypy app
|
|
cd web && npm run check
|
|
|
|
data: ## download the public demo slice: GIAB HG002 + ClinVar, chr22 (see docs/data.md)
|
|
scripts/fetch-demo-data.sh
|
|
|
|
hpo: ## load HPO annotations, propagated up the ontology and weighted by information content
|
|
cd ml && uv run --extra db python ../scripts/load-hpo.py
|
|
|
|
demo-case: ## build the simulated proband: GIAB background + one ClinVar pathogenic variant
|
|
scripts/make-demo-case.sh
|
|
|
|
published-case: ## build a case from a published patient: a GA4GH phenopacket + GIAB background
|
|
scripts/make-published-case.py
|
|
|
|
training-set: ## build a ClinVar training table, shaped like VEP --tab output
|
|
scripts/make-training-set.sh
|
|
|
|
benchmark: ## measure the phenotype ranking against every published case (see docs/data.md)
|
|
docker exec $(DB_CONTAINER) psql -U rarelens -d rarelens -At -F',' \
|
|
-c "select gene_symbol, hpo_id from gene_phenotypes" \
|
|
| tr ',' '\t' > $(BENCH_DIR)/gene_phenotypes.tsv
|
|
test -f $(BENCH_DIR)/all_phenopackets.zip || curl -sL -o $(BENCH_DIR)/all_phenopackets.zip \
|
|
"$$(curl -s https://api.github.com/repos/monarch-initiative/phenopacket-store/releases/latest \
|
|
| sed -n 's/.*"browser_download_url": "\(.*all_phenopackets.zip\)".*/\1/p')"
|
|
cd ml && uv run --extra dev python -m rarelens_ml.benchmark \
|
|
--phenopackets ../$(BENCH_DIR)/all_phenopackets.zip \
|
|
--annotations ../$(BENCH_DIR)/gene_phenotypes.tsv
|
|
|
|
train: ## train the pathogenicity model and point the production alias at it (needs `make up`)
|
|
cd ml && MLFLOW_TRACKING_URI=$(MLFLOW_URI) uv run --extra dev \
|
|
python -m rarelens_ml.train --tsv ../data/clinvar-training.vep.tsv --register
|
|
|
|
loader:
|
|
docker build -t rarelens/loader:dev -f pipeline/loader.Dockerfile pipeline
|
|
|
|
pipeline: loader ## dry run: annotate $(VCF) without touching the database
|
|
cd pipeline && nextflow run main.nf -profile docker --vcf ../$(VCF)
|
|
|
|
annotate: loader ## make annotate JOB=<job id> [VCF=data/x.vcf.gz]
|
|
@test -n "$(JOB)" || (echo "usage: make annotate JOB=<job id> [VCF=...]"; exit 1)
|
|
cd pipeline && DATABASE_URL=$(HOST_DB_URL) nextflow run main.nf -profile docker \
|
|
--vcf ../$(VCF) --job_id $(JOB)
|
|
|
|
images:
|
|
docker build -t rarelens-api:dev api
|
|
docker build -t rarelens-web:dev web
|
|
|
|
kind: images
|
|
kind create cluster --name rarelens 2>/dev/null || true
|
|
kind load docker-image rarelens-api:dev rarelens-web:dev --name rarelens
|
|
kubectl apply -k infra/k8s/overlays/local
|
|
kubectl -n rarelens rollout status deploy/postgres deploy/api deploy/web
|
|
@echo "kubectl -n rarelens port-forward svc/web 8080:80 (UI)"
|
|
@echo "kubectl -n rarelens port-forward svc/api 8000:80 (API, used by the UI)"
|
|
|
|
serverless-deploy: ## deploy the Cloud Run track: make serverless-deploy PROJECT=<id> [TAG=<sha>]
|
|
@test -n "$(PROJECT)" || (echo "usage: make serverless-deploy PROJECT=<gcp project id> [TAG=<image tag>]"; exit 1)
|
|
@test -n "$$TF_VAR_database_url" || echo "note: TF_VAR_database_url is unset; add -var deploy_cloud_sql=true or export a Postgres URL"
|
|
cd infra/terraform && terraform apply -var project=$(PROJECT) -var image_tag=$(TAG)
|
|
|
|
serverless-destroy: ## tear it all down
|
|
cd infra/terraform && terraform destroy -var project=$(PROJECT) -var deletion_protection=false
|
|
|
|
gcp-configure: ## one-time: write your GCP project id into the gcp overlay and Argo manifests
|
|
@test -n "$(PROJECT)" || (echo "usage: make gcp-configure PROJECT=<gcp project id>"; exit 1)
|
|
grep -rl __GCP_PROJECT__ infra/k8s/overlays/gcp infra/argo-workflows \
|
|
| xargs sed -i.bak "s/__GCP_PROJECT__/$(PROJECT)/g"
|
|
find infra -name '*.bak' -delete
|
|
|
|
gcp-secrets: ## after terraform apply: copy DB URLs from Secret Manager into k8s secrets
|
|
@test -n "$(PROJECT)" || (echo "usage: make gcp-secrets PROJECT=<gcp project id>"; exit 1)
|
|
kubectl -n rarelens create secret generic api-secrets --dry-run=client -o yaml \
|
|
--from-literal=DATABASE_URL="$$(gcloud secrets versions access latest --project $(PROJECT) --secret rarelens-api-database-url)" \
|
|
| kubectl apply -f -
|
|
kubectl -n rarelens create secret generic pipeline-secrets --dry-run=client -o yaml \
|
|
--from-literal=DATABASE_URL="$$(gcloud secrets versions access latest --project $(PROJECT) --secret DATABASE_URL)" \
|
|
| kubectl apply -f -
|