# rarelens

A small, end-to-end variant interpretation platform for rare genetic disease research.
Scientists upload a VCF, a Nextflow workflow annotates it with Ensembl VEP, a machine
learning model scores each variant, and results are browsable in a web app.

This repository is a **self-training lab**. It exists so that one engineer can learn, in
public, how a modern life-sciences platform is built end to end: full-stack application,
scientific pipeline, ML serving, and cloud infrastructure, all in one monorepo. It is not a
clinical tool and makes no diagnostic claims.

## What is in the box

| Layer      | Technology                                             | Directory            |
|------------|--------------------------------------------------------|----------------------|
| Pipeline   | Nextflow DSL2, bcftools, Ensembl VEP, Docker, Google Batch | `pipeline/`      |
| API        | FastAPI, Pydantic v2, SQLAlchemy 2.0 (async), Alembic  | `api/`               |
| Database   | PostgreSQL 16                                          | `docker-compose.yml` |
| Frontend   | SvelteKit, TypeScript                                  | `web/`               |
| ML         | LightGBM pathogenicity scorer, MLflow registry         | `ml/`                |
| Orchestration | Argo Workflows + Argo Events (pipeline), Pub/Sub (events) | `infra/argo-workflows/` |
| Platform   | Kubernetes (Kustomize), ArgoCD (GitOps)                | `infra/k8s/`, `infra/argocd/` |
| Cloud      | GCP: GKE Autopilot, Cloud SQL, GCS, Batch, Secret Manager, Artifact Registry | `infra/terraform/` |
| CI/CD      | GitHub Actions, Workload Identity Federation           | `.github/workflows/` |

## Quick start (local)

```bash
make up          # postgres + api + web + mlflow via docker-compose
make migrate     # alembic upgrade head
make data        # real public data: GIAB HG002 + ClinVar, chr22 (needs bcftools)
make test        # api, ml, loader and web tests (no Docker needed for the DB tests)
```

Then open http://localhost:5173.

The docker-compose API has no Nextflow, so "Run VEP annotation" marks the job failed with the
command to run instead. With Nextflow and Docker on the host, a VEP cache in `pipeline/cache/vep`
and a VCF under `data/` (see [data/README.md](data/README.md)):

```bash
make annotate JOB=<job id from the UI> VCF=data/example.vcf.gz
make pipeline VCF=data/example.vcf.gz   # dry run: annotate without touching the database
```

To train and register a model (the API scores with `models:/rarelens-pathogenicity@production`):

```bash
cd ml && MLFLOW_TRACKING_URI=http://localhost:5000 \
  uv run python -m rarelens_ml.train --tsv ../pipeline/results/<sample>.vep.tsv --register
```

Local Kubernetes: `make kind` builds the images, loads them into a kind cluster and applies
`infra/k8s/overlays/local`.

## Deploying to GCP

Two tracks, same code. The serverless one is the default because it costs about £1/month idle;
[docs/cloud.md](docs/cloud.md) has the numbers.

**Serverless (Cloud Run + Google Batch).** The API and the UI scale to zero, and the Nextflow
driver runs as a Cloud Run job only while a pipeline is running.

```bash
cd infra/terraform
terraform init -backend-config="bucket=<tfstate bucket>"
export TF_VAR_database_url='postgresql+asyncpg://user:pass@host/db?sslmode=require'  # e.g. Neon's free tier
terraform apply -var project=<project id>          # add -var deploy_cloud_sql=true to use Cloud SQL instead
cd ../.. && make serverless-deploy PROJECT=<project id> TAG=<commit sha>   # redeploy a new build
```

`terraform output web_url` is the URL to share; it serves the UI and proxies `/api` to the API, so
there is one public address and no CORS. Upload the VEP cache to
`gs://<project>-rarelens-data/refs/vep` before running a real annotation, and set
`-var model_uri=gs://<project>-rarelens-data/models/pathogenicity/1` to score without running an
MLflow server. Set a billing budget first — the demo has no authentication.

**Kubernetes (GKE + Argo + ArgoCD).** Off by default; turn it on to demonstrate the GitOps path,
then destroy it.

```bash
terraform apply -var project=<project id> -var deploy_kubernetes=true -var deploy_cloud_sql=true
make gcp-configure PROJECT=<project id>   # once; commit the result
make gcp-secrets PROJECT=<project id>
```

Then install Argo Workflows, Argo Events and ArgoCD, and `kubectl apply -f infra/argocd/app.yaml`.
Every green CI run on `main` bumps image tags in the gcp overlay and ArgoCD deploys them.
`make serverless-destroy PROJECT=<project id>` tears everything down.

## Data

The demo runs on published, openly licensed human data: the NIST Genome in a Bottle HG002
benchmark genome as the sample, ClinVar for labels, gnomAD for allele frequencies. Sources,
licences, citations and how the model should be evaluated honestly are in
[docs/data.md](docs/data.md).

## Architecture

See [docs/architecture.md](docs/architecture.md) for the diagram and the reasoning behind
each choice, and [docs/cloud.md](docs/cloud.md) for why this deploys to Google Cloud rather
than AWS.

## Status

Work in progress. Milestones, in order:

1. Skeleton, Postgres, FastAPI, Nextflow VEP annotation on a public VCF, CI green
2. SvelteKit UI: sample list, variant table with filters, job status
3. Kubernetes manifests, kind, Argo Workflows trigger
4. Terraform for GCP, ArgoCD GitOps deploy
5. Pathogenicity model, MLflow registry, prediction endpoint

## Licence

AGPL-3.0. Test data are public (ClinVar, gnomAD subsets); no patient data are used or accepted.