Kemal Yaylali
11fb6b3d73
An end-to-end audit found the repo could not build, test or run as shipped. This fixes every finding, then adds a Cloud Run track so the demo costs about £1/month idle instead of ~£150. CI (red on its first run) - api: setuptools could not build the package (flat layout with app/ and alembic/) - web: missing @types/node; `vitest run` exited 1 with no test files - pipeline: the stub run needed a gitignored VCF, and no process had a stub block - ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI) ML serving (scores were meaningless) - the registered model now carries its own feature engineering and returns predict_proba, so serving sends raw columns and cannot drift from training - resolve by registry alias (stages are deprecated in MLflow 3) and record the real version; re-scoring upserts instead of failing on the unique constraint - ClinVar labels parsed from VEP's lowercase terms Pipeline - exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent - job status reaches running/failed/succeeded, so the UI stops polling dead jobs - DATABASE_URL travels in the environment or a Nextflow secret, never on a command line - VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch Deployment - the API serves /api (matching the ingress); the web app reads its API URL at runtime - migrations run in an init container under a Postgres advisory lock - terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload Identity bindings, Secret Manager, deletion protection - serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md. Correctness and security - 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider VEP text columns, enum dropped on downgrade, the sample's assembly actually used - vcf_uri restricted to gs:// objects or files under the data root, blocking option injection - CORS restricted to configured origins; `make down` no longer deletes volumes Data - docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar, gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform validate and both kustomize overlays clean.
113 lines
7.0 KiB
Markdown
113 lines
7.0 KiB
Markdown
# Cloud choice: Google Cloud, with a documented AWS escape hatch
|
||
|
||
Decided 2026-09-12. Scope: `infra/terraform/`, `infra/k8s/`, `pipeline/nextflow.config`.
|
||
|
||
## Decision
|
||
|
||
rarelens deploys to **Google Cloud**. AWS was the serious alternative, and it is genuinely better
|
||
on two points (below), but not by enough to justify rebuilding an estate that already works.
|
||
|
||
## Why Google Cloud
|
||
|
||
| Reason | Detail |
|
||
|---|---|
|
||
| One Kubernetes control plane is effectively free | GKE's free tier gives $74.40/month in credits per billing account, which covers one Autopilot or zonal cluster. EKS charges $0.10/hour per cluster (~$73/month) with no equivalent credit. For a self-funded lab this is the largest fixed monthly difference. |
|
||
| The executor question is already settled here | Google retired Cloud Life Sciences on 8 July 2025; Batch is its successor, and Nextflow upstream moved to Google Batch in April 2025. `pipeline/nextflow.config` uses `google-batch`, which is the supported path rather than a legacy one. |
|
||
| The estate exists and is verified | Terraform (custom VPC, private Cloud SQL, Workload Identity, Secret Manager, Batch IAM), Kustomize overlays, Argo Workflows/Events and CI all render, validate and pass tests today. Rebuilding this on AWS costs 1–2 weeks and mostly repeats learning already banked. |
|
||
| No data lock-in | Every dataset the platform uses is readable from either cloud (see [data.md](data.md)): gnomAD publishes to GCP, AWS and Azure; GIAB and 1000 Genomes are open on AWS and NCBI; ClinVar is a plain NCBI download. |
|
||
|
||
## What AWS is genuinely better at
|
||
|
||
- **Managed Nextflow.** AWS HealthOmics runs Nextflow (up to 26.04), WDL and CWL as a managed
|
||
service, and is available in London (`eu-west-2`). GCP has no equivalent: you operate the
|
||
driver yourself, which is exactly what `infra/argo-workflows/annotate.yaml` does.
|
||
- **UK life-sciences gravity.** The UK Biobank Research Analysis Platform is DNAnexus running on
|
||
AWS, hosted in the UK. If the aim is to mirror what Cambridge-area employers run day to day,
|
||
AWS is the more common answer.
|
||
|
||
## When to revisit this
|
||
|
||
Move to AWS if any of these becomes true:
|
||
|
||
- The lab wants a managed pipeline runner instead of an Argo + Batch driver it maintains.
|
||
- Matching an AWS-first employer's stack matters more than the two weeks it costs.
|
||
- The shape changes: several clusters, or enough managed-service spend that one free control
|
||
plane stops being material.
|
||
|
||
## Running this on a hobby budget
|
||
|
||
The cloud is not the cost driver; the always-on shape is. Estimates below are list price, and
|
||
rounded — treat them as orders of magnitude, not quotes.
|
||
|
||
### What the Kubernetes estate costs at rest
|
||
|
||
GKE Autopilot bills what pods *request*, not what they use, with a per-pod floor (250m vCPU /
|
||
512 MiB). The free tier credit covers the cluster fee only, not pod-hours.
|
||
|
||
| Always-on | Requests | ~Monthly (us-central1 rates: $0.0445/vCPU-h, $0.0049/GiB-h) |
|
||
|---|---|---|
|
||
| api + web (2 replicas each, incl. Cloud SQL proxy sidecar) | ~1.2 vCPU, ~2.3 GiB | ~$47 |
|
||
| ArgoCD, Argo Workflows, Argo Events + NATS EventBus (~11 pods at the floor) | ~2.8 vCPU, ~5.5 GiB | ~$110 |
|
||
| Cloud SQL `db-f1-micro` | — | ~$8–12 |
|
||
| **Total** | | **~$165–170, London a bit more** |
|
||
|
||
That is the wrong shape for a portfolio that is idle 99% of the time.
|
||
|
||
### The shape that costs ~£1/month
|
||
|
||
This is what `terraform apply` builds by default (`deploy_kubernetes` and `deploy_cloud_sql` are
|
||
both `false`). Kubernetes becomes something you switch on to show, not something you rent:
|
||
|
||
| Piece | Service | Idle cost |
|
||
|---|---|---|
|
||
| api, web | Cloud Run, `min-instances=0`, `max_instances` capped | £0 — Always Free covers 2M requests, 180k vCPU-s, 360k GiB-s per month |
|
||
| Nextflow driver | Cloud Run **job**, started by the API through the Jobs API (`roles/run.jobsExecutorWithOverrides`, one job only) | £0 idle, pennies per run |
|
||
| Pipeline tasks | Google Batch on **Spot** VMs | £0 idle; a chr22 VEP run is a few pence |
|
||
| Database | Neon free tier (scale-to-zero, 0.5 GB) via `TF_VAR_database_url`, or `-var deploy_cloud_sql=true` | £0 (or ~$8–12) |
|
||
| Model | pyfunc artifact loaded straight from GCS (`MODEL_URI`), no MLflow server running | £0 |
|
||
| Storage | GCS + Artifact Registry | ~£1 (VEP cache dominates; Nearline halves it) |
|
||
|
||
Trade-offs worth knowing: Cloud Run cold starts add 1–3 s to the first request after idle; the
|
||
Cloud Run path drops Pub/Sub, Argo Events and Argo Workflows from the critical path (the API calls
|
||
the Jobs API directly); and 0.5 GB of Neon does not fit a whole chromosome once `annotations`
|
||
stores the full VEP record — demo a gene panel, or store only the annotation keys the UI uses.
|
||
|
||
Only the web service needs to be public: it serves the UI and proxies `/api` to the API service
|
||
(`web/src/routes/api/[...path]`), which is the same shape the ingress gives the Kubernetes track,
|
||
so the frontend code is identical either way.
|
||
|
||
### Keep the Kubernetes story, stop paying rent for it
|
||
|
||
`infra/k8s/` and `infra/argo-workflows/` stay in the repo and stay deployable. Bring the estate up
|
||
with `terraform apply` for an interview or a recording (roughly $0.25/hour while running, so a
|
||
two-hour demo is small change), then `terraform destroy -var deletion_protection=false`. `make kind`
|
||
runs the same manifests locally for free.
|
||
|
||
### Guardrails
|
||
|
||
- A billing budget with alerts at £5/£10, before anything else.
|
||
- `max-instances` on every Cloud Run service: scale-to-zero protects the floor, a cap protects the ceiling.
|
||
- Spot VMs for Batch, and the existing 30-day lifecycle rule on `work/` in the bucket.
|
||
- New accounts get $300 of Google Cloud credit for 90 days, which covers the experimenting phase.
|
||
|
||
### One more reason not to switch to AWS
|
||
|
||
AWS replaced its 12-month free tier on 15 July 2025 with credits ($100, up to $200) on a Free plan
|
||
that closes after six months or when the credits run out. Google's Always Free quotas, including
|
||
Cloud Run's, are permanent. For a demo meant to stay reachable indefinitely at near-zero cost,
|
||
that difference matters more than any feature comparison above.
|
||
|
||
## The escape hatch
|
||
|
||
Nextflow is the portability layer: executors are configuration, not code. An AWS run needs a new
|
||
profile in `pipeline/nextflow.config` (`process.executor = 'awsbatch'`, an S3 work directory and a
|
||
job queue), or a HealthOmics workflow definition. The processes themselves do not change. Keeping
|
||
`pipeline/bin/` cloud-agnostic (the scripts read `DATABASE_URL` from the environment, never from a
|
||
command line) is what keeps that true.
|
||
|
||
Sources: [Migrate to Batch from Cloud Life Sciences](https://docs.cloud.google.com/batch/docs/migrate-to-batch-from-cloud-life-sciences),
|
||
[GKE pricing](https://cloud.google.com/kubernetes-engine/pricing),
|
||
[HealthOmics supported languages](https://docs.aws.amazon.com/omics/latest/dev/workflows-supported-languages.html),
|
||
[HealthOmics Nextflow 26.04](https://aws.amazon.com/about-aws/whats-new/2026/06/aws-healthomics-nextflow-version-26-04/),
|
||
[UK Biobank Research Analysis Platform](https://www.ukbiobank.ac.uk/use-our-data/research-analysis-platform/).
|