rarelens

A small, end-to-end variant interpretation platform for rare genetic disease research. Scientists upload a VCF, a Nextflow workflow annotates it with Ensembl VEP, a machine learning model scores each variant, and results are browsable in a web app.

This repository is a self-training lab. It exists so that one engineer can learn, in public, how a modern life-sciences platform is built end to end: full-stack application, scientific pipeline, ML serving, and cloud infrastructure, all in one monorepo. It is not a clinical tool and makes no diagnostic claims.

What is in the box

Layer	Technology	Directory
Pipeline	Nextflow DSL2, bcftools, Ensembl VEP, Docker, Google Batch	pipeline/
API	FastAPI, Pydantic v2, SQLAlchemy 2.0 (async), Alembic	api/
Database	PostgreSQL 16	docker-compose.yml
Frontend	SvelteKit, TypeScript	web/
ML	LightGBM pathogenicity scorer, MLflow registry	ml/
Orchestration	Argo Workflows + Argo Events (pipeline), Pub/Sub (events)	infra/argo-workflows/
Platform	Kubernetes (Kustomize), ArgoCD (GitOps)	infra/k8s/, infra/argocd/
Cloud	GCP: GKE Autopilot, Cloud SQL, GCS, Batch, Secret Manager, Artifact Registry	infra/terraform/
CI/CD	GitHub Actions, Workload Identity Federation	.github/workflows/

Quick start (local)

make up          # postgres + api + web + mlflow via docker-compose
make migrate     # alembic upgrade head
make data        # real public data: GIAB HG002 + ClinVar, chr22 (needs bcftools)
make test        # api, ml, loader and web tests (no Docker needed for the DB tests)

Then open http://localhost:5173.

The docker-compose API has no Nextflow, so "Run VEP annotation" marks the job failed with the command to run instead. With Nextflow and Docker on the host, a VEP cache in pipeline/cache/vep and a VCF under data/ (see data/README.md):

make annotate JOB=<job id from the UI> VCF=data/example.vcf.gz
make pipeline VCF=data/example.vcf.gz   # dry run: annotate without touching the database

No cache? VEP_DATABASE=true queries Ensembl's public database instead. It is slow per variant and returns fewer fields, but it needs no 25 GB download, which is enough to demonstrate the pipeline on a handful of variants:

VEP_DATABASE=true make pipeline VCF=pipeline/tests/data/tiny.vcf

To make the UI's "Run VEP annotation" button work, run the API on the host (where Nextflow and Docker are) rather than in docker-compose:

docker compose up -d db
cd api && DATABASE_URL=postgresql+asyncpg://rarelens:rarelens@localhost:5432/rarelens \
  PIPELINE_DATABASE_URL=postgresql+asyncpg://rarelens:[email protected]:5432/rarelens \
  LOCAL_DATA_ROOT=$PWD/.. VEP_DATABASE=true \
  uv run --extra dev uvicorn app.main:app --port 8000

PIPELINE_DATABASE_URL is what the loader container gets: inside it, the API's own localhost would be the container itself. LOCAL_DATA_ROOT is the directory a sample's vcf_uri must sit under.

To train and register a model (the API scores with models:/rarelens-pathogenicity@production):

cd ml && MLFLOW_TRACKING_URI=http://localhost:5000 \
  uv run python -m rarelens_ml.train --tsv ../pipeline/results/<sample>.vep.tsv --register

Local Kubernetes: make kind builds the images, loads them into a kind cluster and applies infra/k8s/overlays/local.

Deploying to GCP

Two tracks, same code. The serverless one is the default because it costs about £1/month idle; docs/cloud.md has the numbers.

Serverless (Cloud Run + Google Batch). The API and the UI scale to zero, and the Nextflow driver runs as a Cloud Run job only while a pipeline is running.

cd infra/terraform
terraform init -backend-config="bucket=<tfstate bucket>"
export TF_VAR_database_url='postgresql+asyncpg://user:pass@host/db?sslmode=require'  # e.g. Neon's free tier
terraform apply -var project=<project id>          # add -var deploy_cloud_sql=true to use Cloud SQL instead
cd ../.. && make serverless-deploy PROJECT=<project id> TAG=<commit sha>   # redeploy a new build

terraform output web_url is the URL to share; it serves the UI and proxies /api to the API, so there is one public address and no CORS. Upload the VEP cache to gs://<project>-rarelens-data/refs/vep before running a real annotation, and set -var model_uri=gs://<project>-rarelens-data/models/pathogenicity/1 to score without running an MLflow server. Set a billing budget first — the demo has no authentication.

Kubernetes (GKE + Argo + ArgoCD). Off by default; turn it on to demonstrate the GitOps path, then destroy it.

terraform apply -var project=<project id> -var deploy_kubernetes=true -var deploy_cloud_sql=true
make gcp-configure PROJECT=<project id>   # once; commit the result
make gcp-secrets PROJECT=<project id>

Then install Argo Workflows, Argo Events and ArgoCD, and kubectl apply -f infra/argocd/app.yaml. Every green CI run on main bumps image tags in the gcp overlay and ArgoCD deploys them. make serverless-destroy PROJECT=<project id> tears everything down.

Data

The demo runs on published, openly licensed human data: the NIST Genome in a Bottle HG002 benchmark genome as the sample, ClinVar for labels, gnomAD for allele frequencies. Sources, licences, citations and how the model should be evaluated honestly are in docs/data.md.

Architecture

See docs/architecture.md for the diagram and the reasoning behind each choice, and docs/cloud.md for why this deploys to Google Cloud rather than AWS.

Status

Work in progress. Milestones, in order:

1. Skeleton, Postgres, FastAPI, Nextflow VEP annotation on a public VCF, CI green
2. SvelteKit UI: sample list, variant table with filters, job status
3. Kubernetes manifests, kind, Argo Workflows trigger
4. Terraform for GCP, ArgoCD GitOps deploy
5. Pathogenicity model, MLflow registry, prediction endpoint

Licence

AGPL-3.0. Test data are public (ClinVar, gnomAD subsets); no patient data are used or accepted.