diff --git a/README.md b/README.md index bd5d2e3..9af1b1f 100644 --- a/README.md +++ b/README.md @@ -21,9 +21,9 @@ clinical tool and makes no diagnostic claims. | Database | PostgreSQL 16 | `docker-compose.yml` | | Frontend | SvelteKit, TypeScript | `web/` | | ML | LightGBM pathogenicity scorer, MLflow registry | `ml/` | -| Orchestration | Argo Workflows + Argo Events (pipeline), Pub/Sub (events) | `infra/argo-workflows/` | +| Orchestration | Cloud Run job (default), Argo Workflows + Argo Events, Pub/Sub | `infra/terraform/`, `infra/argo-workflows/` | | Platform | Kubernetes (Kustomize), ArgoCD (GitOps) | `infra/k8s/`, `infra/argocd/` | -| Cloud | GCP: GKE Autopilot, Cloud SQL, GCS, Batch, Secret Manager, Artifact Registry | `infra/terraform/` | +| Cloud | GCP: Cloud Run, Google Batch, GCS, Secret Manager, Artifact Registry; GKE Autopilot and Cloud SQL behind flags | `infra/terraform/` | | CI/CD | GitHub Actions, Workload Identity Federation | `.github/workflows/` | ## Quick start (local) @@ -51,7 +51,7 @@ the rest of that patient's genome is not public. It writes the phenotype list al so the case can be created exactly as reported. See [docs/data.md](docs/data.md) for the provenance and for what this case does and does not demonstrate. -The docker-compose API has no Nextflow, so "Run VEP annotation" marks the job failed with the +The docker-compose API has no Nextflow, so "Analyse case" marks the job failed with the command to run instead. With Nextflow and Docker on the host, a VEP cache in `pipeline/cache/vep` and a VCF under `data/` (see [data/README.md](data/README.md)): @@ -68,7 +68,7 @@ pipeline on a handful of variants: VEP_DATABASE=true make pipeline VCF=pipeline/tests/data/tiny.vcf ``` -To make the UI's "Run VEP annotation" button work, run the API on the host (where Nextflow and +To make the UI's "Analyse case" button work, run the API on the host (where Nextflow and Docker are) rather than in docker-compose: ```bash @@ -80,7 +80,7 @@ cd api && DATABASE_URL=postgresql+asyncpg://rarelens:rarelens@localhost:5432/rar ``` `PIPELINE_DATABASE_URL` is what the loader container gets: inside it, the API's own `localhost` -would be the container itself. `LOCAL_DATA_ROOT` is the directory a sample's `vcf_uri` must sit under. +would be the container itself. `LOCAL_DATA_ROOT` is the directory a case's `vcf_uri` must sit under. To train and register a model (the API scores with `models:/rarelens-pathogenicity@production`): @@ -133,9 +133,10 @@ Every green CI run on `main` bumps image tags in the gcp overlay and ArgoCD depl ## Data The demo runs on published, openly licensed human data: the NIST Genome in a Bottle HG002 -benchmark genome as the sample, ClinVar for labels, gnomAD for allele frequencies. Sources, -licences, citations and how the model should be evaluated honestly are in -[docs/data.md](docs/data.md). +benchmark genome as the background sample, ClinVar for labels, gnomAD for allele frequencies, the +Human Phenotype Ontology's gene-to-phenotype annotations as what the ranking matches against, and +GA4GH phenopackets curated from case reports for the published case. Sources, licences, citations +and how the model should be evaluated honestly are in [docs/data.md](docs/data.md). ## Architecture @@ -145,13 +146,18 @@ than AWS. ## Status -Work in progress. Milestones, in order: +A self-training lab, built in the open. Working end to end: -1. Skeleton, Postgres, FastAPI, Nextflow VEP annotation on a public VCF, CI green -2. SvelteKit UI: sample list, variant table with filters, job status +1. Postgres, FastAPI, Nextflow VEP annotation on a public VCF, CI green +2. SvelteKit UI: cases, a phenotype-ranked candidate list showing the evidence behind each rank, + shortlist/dismiss decisions, and a case report 3. Kubernetes manifests, kind, Argo Workflows trigger -4. Terraform for GCP, ArgoCD GitOps deploy -5. Pathogenicity model, MLflow registry, prediction endpoint +4. Terraform for GCP: serverless Cloud Run + Batch by default, GKE and ArgoCD behind a flag +5. Pathogenicity model trained on ClinVar, MLflow registry, scoring endpoint +6. A demo case built from a published patient, with the citations behind it + +Known gaps: allele frequencies need the 25 GB VEP cache, because VEP's database mode returns +none, so the rarity term does no work without it; and the deployed demo has no authentication. ## Licence