kemal/rarelens
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

fix: overhaul the platform skeleton, add a serverless deployment track

Browse Source 
An end-to-end audit found the repo could not build, test or run as shipped. This
fixes every finding, then adds a Cloud Run track so the demo costs about £1/month
idle instead of ~£150.

CI (red on its first run)
- api: setuptools could not build the package (flat layout with app/ and alembic/)
- web: missing @types/node; `vitest run` exited 1 with no test files
- pipeline: the stub run needed a gitignored VCF, and no process had a stub block
- ruff pinned, mypy configured, DB tests on real Postgres (pgserver locally, service in CI)

ML serving (scores were meaningless)
- the registered model now carries its own feature engineering and returns predict_proba,
  so serving sends raw columns and cannot drift from training
- resolve by registry alias (stages are deprecated in MLflow 3) and record the real
  version; re-scoring upserts instead of failing on the unique constraint
- ClinVar labels parsed from VEP's lowercase terms

Pipeline
- exact ref/alt recovered from a CHROM_POS_REF_ALT VCF ID; loading is idempotent
- job status reaches running/failed/succeeded, so the UI stops polling dead jobs
- DATABASE_URL travels in the environment or a Nextflow secret, never on a command line
- VEP cache and plugins staged as inputs; the gcp profile runs tasks on Google Batch

Deployment
- the API serves /api (matching the ingress); the web app reads its API URL at runtime
- migrations run in an init container under a Postgres advisory lock
- terraform: custom VPC shared with Batch, private Cloud SQL, API enablement, Workload
  Identity bindings, Secret Manager, deletion protection
- serverless track, now the default: Cloud Run services scaling to zero, a Cloud Run job
  for the Nextflow driver, and Neon or Cloud SQL behind one DATABASE_URL secret. GKE and
  Argo remain, behind -var deploy_kubernetes=true. See docs/cloud.md.

Correctness and security
- 409 on duplicate sample names, 422 on bad paging, natural chromosome ordering, wider
  VEP text columns, enum dropped on downgrade, the sample's assembly actually used
- vcf_uri restricted to gs:// objects or files under the data root, blocking option injection
- CORS restricted to configured origins; `make down` no longer deletes volumes

Data
- docs/data.md records the peer-reviewed, openly licensed sources (GIAB HG002, ClinVar,
  gnomAD) with citations and an honest evaluation plan; `make data` fetches a chr22 slice

Verified: api 50 tests, ml 18, loader 16, web 12; ruff, mypy, svelte-check, terraform
validate and both kustomize overlays clean.
This commit is contained in:
Kemal Yaylali
2026-09-12 07:21:11 +01:00
parent 5463f489a3
commit 11fb6b3d73
100 changed files with 3431 additions and 340 deletions
Download Patch File Download Diff File Expand all files Collapse all files
infra/argo-workflows/annotate.yaml
+35 -18
View File
@@ -1,37 +1,54 @@
# Triggered by an Argo Events sensor listening on the Pub/Sub topic "vcf-uploaded".
# Submitted by the Argo Events sensor in events.yaml for each "vcf-uploaded" Pub/Sub message.
# The Nextflow driver runs here; its tasks run on Google Batch (see the gcp profile in
# pipeline/nextflow.config). Image names are rewritten by the gcp overlay and bumped by CI.
apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata: { name: annotate-vcf, namespace: rarelens }
metadata: { name: annotate-vcf }
spec:
serviceAccountName: rarelens-pipeline
entrypoint: nextflow
onExit: exit-handler
arguments:
parameters:
- { name: job_id }
- { name: vcf_uri }
- { name: assembly, value: GRCh38 }
templates:
- name: nextflow
inputs:
parameters: [{ name: job_id }, { name: vcf_uri }]
serviceAccountName: rarelens-pipeline
container:
image: europe-west2-docker.pkg.dev/PROJECT/rarelens/pipeline:latest
command: [nextflow]
image: rarelens/pipeline
args:
- run
- /pipeline/main.nf
- -profile
- gcp
- --vcf
- "{{inputs.parameters.vcf_uri}}"
- "{{workflow.parameters.vcf_uri}}"
- --job_id
- "{{inputs.parameters.job_id}}"
- --db_url
- "$(DATABASE_URL)"
envFrom: [{ secretRef: { name: api-secrets } }]
resources: { requests: { cpu: "2", memory: 4Gi } }
- name: score
# Optional GPU step for the deep-learning baseline; Autopilot schedules on an L4 node.
nodeSelector: { cloud.google.com/gke-accelerator: nvidia-l4 }
- "{{workflow.parameters.job_id}}"
- --assembly
- "{{workflow.parameters.assembly}}"
# GCP_PROJECT / GCS_BUCKET / GCP_REGION feed params in nextflow.config.
envFrom: [{ configMapRef: { name: pipeline-config } }]
resources: { requests: { cpu: "1", memory: 2Gi } }
# The loader marks success; anything else (Nextflow error, OOM, eviction) is marked here
# so the UI never polls a dead job forever.
- name: exit-handler
steps:
- - name: mark-failed
template: mark-failed
when: "{{workflow.status}} != Succeeded"
- name: mark-failed
container:
image: europe-west2-docker.pkg.dev/PROJECT/rarelens/ml:latest
resources: { limits: { nvidia.com/gpu: 1 } }
image: rarelens/loader
command: [set_job_status.py]
args:
- --job-id
- "{{workflow.parameters.job_id}}"
- --status
- failed
- --log
- "Argo workflow {{workflow.name}} ended {{workflow.status}}"
envFrom: [{ secretRef: { name: pipeline-secrets } }]
resources: { requests: { cpu: 100m, memory: 256Mi } }
infra/argo-workflows/events.yaml
+46
View File
@@ -0,0 +1,46 @@
# Pub/Sub "vcf-uploaded" (published by the API) -> Argo Workflow from the annotate-vcf template.
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata: { name: vcf-uploaded }
spec:
template:
# Workload Identity: roles/pubsub.subscriber on the subscription (infra/terraform/iam.tf).
serviceAccountName: rarelens-pipeline
pubSub:
vcf-uploaded:
projectID: __GCP_PROJECT__
subscriptionID: vcf-uploaded-argo # created by Terraform (infra/terraform/pubsub.tf)
jsonBody: true
---
apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata: { name: annotate-vcf }
spec:
template:
serviceAccountName: argo-events-sensor
dependencies:
- { name: vcf, eventSourceName: vcf-uploaded, eventName: vcf-uploaded }
triggers:
- template:
name: submit-annotate-vcf
argoWorkflow:
operation: submit
source:
resource:
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata: { generateName: annotate-vcf- }
spec:
workflowTemplateRef: { name: annotate-vcf }
arguments:
parameters:
- { name: job_id }
- { name: vcf_uri }
- { name: assembly }
parameters:
- src: { dependencyName: vcf, dataKey: body.job_id }
dest: spec.arguments.parameters.0.value
- src: { dependencyName: vcf, dataKey: body.vcf_uri }
dest: spec.arguments.parameters.1.value
- src: { dependencyName: vcf, dataKey: body.assembly, value: GRCh38 }
dest: spec.arguments.parameters.2.value
infra/argo-workflows/kustomization.yaml
+5
View File
@@ -0,0 +1,5 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# Requires Argo Workflows and Argo Events (with the default EventBus) installed in the cluster.
namespace: rarelens
resources: [rbac.yaml, annotate.yaml, events.yaml]
infra/argo-workflows/rbac.yaml
+41
View File
@@ -0,0 +1,41 @@
# Runs the annotate-vcf workflow pods and the Pub/Sub EventSource. Bound to the rarelens-pipeline
# Google service account through Workload Identity (annotation added by the gcp overlay).
apiVersion: v1
kind: ServiceAccount
metadata: { name: rarelens-pipeline }
---
# Argo's executor reports step outputs through WorkflowTaskResults.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: argo-executor }
rules:
- apiGroups: [argoproj.io]
resources: [workflowtaskresults]
verbs: [create, patch]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: rarelens-pipeline-argo-executor }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: argo-executor }
subjects: [{ kind: ServiceAccount, name: rarelens-pipeline }]
---
apiVersion: v1
kind: ServiceAccount
metadata: { name: argo-events-sensor }
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: { name: workflow-submitter }
rules:
- apiGroups: [argoproj.io]
resources: [workflows]
verbs: [create]
- apiGroups: [argoproj.io]
resources: [workflowtemplates]
verbs: [get, list]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: { name: argo-events-sensor-workflow-submitter }
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: workflow-submitter }
subjects: [{ kind: ServiceAccount, name: argo-events-sensor }]
infra/k8s/base/api.yaml
+12 -1
View File
@@ -8,13 +8,24 @@ spec:
metadata: { labels: { app: api } }
spec:
serviceAccountName: rarelens-api
initContainers:
# alembic/env.py holds a Postgres advisory lock, so replicas starting together migrate once.
- name: migrate
image: rarelens/api
command: [alembic, upgrade, head]
envFrom: [{ secretRef: { name: api-secrets } }]
resources: { requests: { cpu: 100m, memory: 256Mi }, limits: { cpu: 500m, memory: 512Mi } }
containers:
- name: api
image: rarelens/api
ports: [{ containerPort: 8000 }]
envFrom: [{ secretRef: { name: api-secrets } }]
envFrom:
- secretRef: { name: api-secrets }
- configMapRef: { name: api-config }
readinessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 5 }
livenessProbe: { httpGet: { path: /health, port: 8000 }, periodSeconds: 10, failureThreshold: 6 }
resources: { requests: { cpu: 250m, memory: 512Mi }, limits: { cpu: "1", memory: 1Gi } }
securityContext: { allowPrivilegeEscalation: false }
---
apiVersion: v1
kind: Service
infra/k8s/base/web.yaml
+4
View File
@@ -11,8 +11,12 @@ spec:
- name: web
image: rarelens/web
ports: [{ containerPort: 3000 }]
# Read at runtime ($env/dynamic/public); the ingress routes /api to the API service.
env: [{ name: PUBLIC_API_URL, value: /api }]
readinessProbe: { httpGet: { path: /, port: 3000 }, periodSeconds: 5 }
livenessProbe: { httpGet: { path: /, port: 3000 }, periodSeconds: 10, failureThreshold: 6 }
resources: { requests: { cpu: 100m, memory: 128Mi }, limits: { cpu: 500m, memory: 256Mi } }
securityContext: { allowPrivilegeEscalation: false }
---
apiVersion: v1
kind: Service
infra/k8s/overlays/gcp/kustomization.yaml
+39 -7
View File
@@ -1,21 +1,53 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources: [../../base]
# Project-specific values use the __GCP_PROJECT__ placeholder: run `make gcp-configure PROJECT=<id>`
# once and commit. Image tags are then bumped by .github/workflows/deploy.yml after each green CI.
# Generated ConfigMaps must share the Deployments' namespace, or their hashed names are not
# propagated to envFrom references.
namespace: rarelens
resources: [../../base, ../../../argo-workflows]
configurations: [kustomizeconfig.yaml]
images:
- { name: rarelens/api, newName: europe-west2-docker.pkg.dev/PROJECT/rarelens/api, newTag: latest }
- { name: rarelens/web, newName: europe-west2-docker.pkg.dev/PROJECT/rarelens/web, newTag: latest }
- { name: rarelens/api, newName: europe-west2-docker.pkg.dev/__GCP_PROJECT__/rarelens/api, newTag: latest }
- { name: rarelens/web, newName: europe-west2-docker.pkg.dev/__GCP_PROJECT__/rarelens/web, newTag: latest }
- { name: rarelens/pipeline, newName: europe-west2-docker.pkg.dev/__GCP_PROJECT__/rarelens/pipeline, newTag: latest }
- { name: rarelens/loader, newName: europe-west2-docker.pkg.dev/__GCP_PROJECT__/rarelens/loader, newTag: latest }
configMapGenerator:
- name: api-config
literals: [PUBSUB_TOPIC=vcf-uploaded, GCP_PROJECT=__GCP_PROJECT__]
- name: pipeline-config
# Referenced by name from the Argo WorkflowTemplate, which kustomize does not rewrite.
options: { disableNameSuffixHash: true }
literals: [GCP_PROJECT=__GCP_PROJECT__, GCP_REGION=europe-west2, GCS_BUCKET=__GCP_PROJECT__-rarelens-data]
# api-secrets and pipeline-secrets come from Secret Manager: `make gcp-secrets PROJECT=<id>`.
patches:
- target: { kind: ServiceAccount, name: rarelens-api }
patch: |
- op: add
path: /metadata/annotations
value: { iam.gke.io/gcp-service-account: [email protected] }
value: { iam.gke.io/gcp-service-account: rarelens-api@__GCP_PROJECT__.iam.gserviceaccount.com }
- target: { kind: ServiceAccount, name: rarelens-pipeline }
patch: |
- op: add
path: /metadata/annotations
value: { iam.gke.io/gcp-service-account: rarelens-pipeline@__GCP_PROJECT__.iam.gserviceaccount.com }
- target: { kind: Deployment, name: api }
patch: |
- op: add
path: /spec/template/spec/containers/-
path: /spec/template/spec/initContainers/0
value:
name: cloud-sql-proxy
image: gcr.io/cloud-sql-connectors/cloud-sql-proxy:2.13.0
args: ["--structured-logs", "--port=5432", "PROJECT:europe-west2:rarelens-pg"]
securityContext: { runAsNonRoot: true }
# Native sidecar: starts (and passes its startup probe) before the migrate init
# container, then keeps running alongside the API.
restartPolicy: Always
args:
- --structured-logs
- --private-ip
- --port=5432
- --health-check
- --http-address=0.0.0.0
- __GCP_PROJECT__:europe-west2:rarelens-pg
startupProbe: { httpGet: { path: /startup, port: 9090 }, periodSeconds: 1, failureThreshold: 60 }
securityContext: { runAsNonRoot: true, allowPrivilegeEscalation: false }
resources: { requests: { cpu: 100m, memory: 128Mi }, limits: { cpu: 500m, memory: 256Mi } }
infra/k8s/overlays/gcp/kustomizeconfig.yaml
+4
View File
@@ -0,0 +1,4 @@
# Teach the `images:` transformer where Argo WorkflowTemplates keep their images.
images:
- path: spec/templates/container/image
kind: WorkflowTemplate
infra/k8s/overlays/local/kustomization.yaml
+17
View File
@@ -1,5 +1,8 @@
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
# `make kind` builds and loads the dev images, then applies this overlay. There is no ingress in
# kind: port-forward svc/web to 8080 and svc/api to 8000 (the UI calls the API directly).
namespace: rarelens # generated Secret/ConfigMap names only propagate within one namespace
resources: [../../base, postgres.yaml]
images:
- { name: rarelens/api, newName: rarelens-api, newTag: dev }
@@ -7,3 +10,17 @@ images:
secretGenerator:
- name: api-secrets
literals: [DATABASE_URL=postgresql+asyncpg://rarelens:rarelens@postgres:5432/rarelens]
configMapGenerator:
- name: api-config
literals: ['CORS_ORIGINS=["http://localhost:8080"]']
patches:
- patch: |
apiVersion: apps/v1
kind: Deployment
metadata: { name: web }
spec:
template:
spec:
containers:
- name: web
env: [{ name: PUBLIC_API_URL, value: "http://localhost:8000/api" }]
infra/k8s/overlays/local/postgres.yaml
+2
View File
@@ -13,6 +13,8 @@ spec:
- { name: POSTGRES_USER, value: rarelens }
- { name: POSTGRES_PASSWORD, value: rarelens }
- { name: POSTGRES_DB, value: rarelens }
readinessProbe: { exec: { command: [pg_isready, -U, rarelens] }, periodSeconds: 5 }
resources: { requests: { cpu: 100m, memory: 256Mi }, limits: { cpu: 500m, memory: 512Mi } }
---
apiVersion: v1
kind: Service
infra/terraform/apis.tf
+18
View File
@@ -0,0 +1,18 @@
resource "google_project_service" "enabled" {
for_each = toset([
"artifactregistry.googleapis.com",
"batch.googleapis.com",
"compute.googleapis.com",
"container.googleapis.com",
"iam.googleapis.com",
"iamcredentials.googleapis.com",
"logging.googleapis.com",
"pubsub.googleapis.com",
"secretmanager.googleapis.com",
"servicenetworking.googleapis.com",
"sqladmin.googleapis.com",
"sts.googleapis.com",
])
service = each.value
disable_on_destroy = false
}
infra/terraform/cloudrun.tf
+178
View File
@@ -0,0 +1,178 @@
# The serverless track: scale-to-zero services and an on-demand pipeline driver.
# Idle cost is storage only; see docs/cloud.md.
resource "google_cloud_run_v2_service" "api" {
name = "rarelens-api"
location = var.region
deletion_protection = false
ingress = "INGRESS_TRAFFIC_ALL"
template {
service_account = google_service_account.api.email
scaling {
min_instance_count = 0 # nothing runs, and nothing is billed, between visits
max_instance_count = var.max_instances
}
containers {
image = "${local.registry}/api:${var.image_tag}"
ports { container_port = 8000 }
resources {
limits = { cpu = "1", memory = "1Gi" }
cpu_idle = true # bill CPU only while a request is in flight
startup_cpu_boost = true
}
env {
name = "DATABASE_URL"
value_source {
secret_key_ref {
secret = google_secret_manager_secret.api_database_url.secret_id
version = "latest"
}
}
}
env {
name = "CLOUDRUN_JOB"
value = google_cloud_run_v2_job.nextflow.name
}
env {
name = "GCP_PROJECT"
value = var.project
}
env {
name = "GCP_REGION"
value = var.region
}
env {
name = "GCS_BUCKET"
value = google_storage_bucket.data.name
}
env {
name = "MODEL_URI"
value = var.model_uri
}
}
}
depends_on = [google_secret_manager_secret_version.api_database_url]
}
resource "google_cloud_run_v2_service" "web" {
name = "rarelens-web"
location = var.region
deletion_protection = false
ingress = "INGRESS_TRAFFIC_ALL"
template {
scaling {
min_instance_count = 0
max_instance_count = var.max_instances
}
containers {
image = "${local.registry}/web:${var.image_tag}"
ports { container_port = 3000 }
resources {
limits = { cpu = "1", memory = "512Mi" }
cpu_idle = true
startup_cpu_boost = true
}
# The browser calls /api on this origin; src/routes/api/[...path] forwards it, so there is
# one public URL and no CORS, exactly as the ingress arranges in the Kubernetes track.
env {
name = "PUBLIC_API_URL"
value = "/api"
}
env {
name = "API_INTERNAL_URL"
value = google_cloud_run_v2_service.api.uri
}
# adapter-node sits behind Cloud Run's proxy; derive the origin from the forwarded headers.
env {
name = "PROTOCOL_HEADER"
value = "x-forwarded-proto"
}
env {
name = "HOST_HEADER"
value = "x-forwarded-host"
}
}
}
}
# The Nextflow driver. Started per annotation by the API (overriding the container args); the
# pipeline's own tasks then run on Google Batch (the gcp profile in pipeline/nextflow.config).
resource "google_cloud_run_v2_job" "nextflow" {
name = "rarelens-nextflow"
location = var.region
deletion_protection = false
template {
task_count = 1
template {
service_account = google_service_account.pipeline.email
max_retries = 0
timeout = "7200s"
containers {
image = "${local.registry}/pipeline:${var.image_tag}"
args = ["-version"] # replaced on every execution by the API's overrides
resources {
limits = { cpu = "1", memory = "2Gi" }
}
env {
name = "GCP_PROJECT"
value = var.project
}
env {
name = "GCP_REGION"
value = var.region
}
env {
name = "GCS_BUCKET"
value = google_storage_bucket.data.name
}
env {
name = "NXF_ANSI_LOG"
value = "false"
}
}
}
}
}
# Anyone can open the UI and the API. There is no authentication by design (docs/architecture.md);
# max_instances and a billing budget are what bound the cost.
resource "google_cloud_run_v2_service_iam_member" "web_public" {
project = var.project
location = google_cloud_run_v2_service.web.location
name = google_cloud_run_v2_service.web.name
role = "roles/run.invoker"
member = "allUsers"
}
resource "google_cloud_run_v2_service_iam_member" "api_public" {
project = var.project
location = google_cloud_run_v2_service.api.location
name = google_cloud_run_v2_service.api.name
role = "roles/run.invoker"
member = "allUsers"
}
# Least privilege: the API may execute this one job with argument overrides, nothing more.
resource "google_cloud_run_v2_job_iam_member" "api_runs_nextflow" {
project = var.project
location = google_cloud_run_v2_job.nextflow.location
name = google_cloud_run_v2_job.nextflow.name
role = "roles/run.jobsExecutorWithOverrides"
member = "serviceAccount:${google_service_account.api.email}"
}
resource "google_secret_manager_secret_iam_member" "api_database_url" {
secret_id = google_secret_manager_secret.api_database_url.secret_id
role = "roles/secretmanager.secretAccessor"
member = "serviceAccount:${google_service_account.api.email}"
}
# Reading the model artifact from gs://<bucket>/models/... when MODEL_URI is set.
resource "google_storage_bucket_iam_member" "api_reads_data" {
bucket = google_storage_bucket.data.name
role = "roles/storage.objectViewer"
member = "serviceAccount:${google_service_account.api.email}"
}
infra/terraform/cloudsql.tf
+16 -8
View File
@@ -1,8 +1,10 @@
resource "google_sql_database_instance" "pg" {
name = "rarelens-pg"
database_version = "POSTGRES_16"
region = var.region
deletion_protection = false
count = var.deploy_cloud_sql ? 1 : 0
name = "rarelens-pg"
database_version = "POSTGRES_16"
region = var.region
deletion_protection = var.deletion_protection
depends_on = [google_service_networking_connection.private_services]
settings {
tier = "db-f1-micro" # lab budget; bump for real use
@@ -16,14 +18,20 @@ resource "google_sql_database_instance" "pg" {
}
resource "google_sql_database" "rarelens" {
count = var.deploy_cloud_sql ? 1 : 0
name = "rarelens"
instance = google_sql_database_instance.pg.name
instance = google_sql_database_instance.pg[0].name
}
resource "google_sql_user" "api" {
count = var.deploy_cloud_sql ? 1 : 0
name = "rarelens"
instance = google_sql_database_instance.pg.name
password = random_password.pg.result
instance = google_sql_database_instance.pg[0].name
password = random_password.pg[0].result
}
resource "random_password" "pg" { length = 32 }
resource "random_password" "pg" {
count = var.deploy_cloud_sql ? 1 : 0
length = 32
special = false # embedded in DATABASE_URL, where characters like @ / # % would break parsing
}
infra/terraform/gke.tf
+9 -4
View File
@@ -1,8 +1,13 @@
resource "google_container_cluster" "rarelens" {
name = "rarelens"
location = var.region
enable_autopilot = true
deletion_protection = false
count = var.deploy_kubernetes ? 1 : 0
name = "rarelens"
location = var.region
enable_autopilot = true
deletion_protection = var.deletion_protection
# Same VPC as Cloud SQL's private IP; without this the cluster lands on the "default" network.
network = google_compute_network.vpc.id
subnetwork = google_compute_subnetwork.gke.id
ip_allocation_policy {}
workload_identity_config { workload_pool = "${var.project}.svc.id.goog" }
release_channel { channel = "REGULAR" }
infra/terraform/iam.tf
+52 -7
View File
@@ -1,6 +1,7 @@
# Workload Identity Federation: GitHub Actions pushes images without long-lived keys.
resource "google_iam_workload_identity_pool" "github" {
workload_identity_pool_id = "github"
depends_on = [google_project_service.enabled]
}
resource "google_iam_workload_identity_pool_provider" "github" {
@@ -29,22 +30,66 @@ resource "google_artifact_registry_repository_iam_member" "ci_push" {
member = "serviceAccount:${google_service_account.ci.email}"
}
# Runtime identities (bound to k8s ServiceAccounts via GKE Workload Identity)
resource "google_service_account" "api" { account_id = "rarelens-api" }
# Runtime identities, bound to Kubernetes ServiceAccounts in namespace "rarelens" via GKE
# Workload Identity (the gcp overlay annotates the k8s side).
resource "google_service_account" "api" { account_id = "rarelens-api" }
resource "google_service_account" "pipeline" { account_id = "rarelens-pipeline" }
resource "google_service_account_iam_member" "api_workload_identity" {
count = var.deploy_kubernetes ? 1 : 0
service_account_id = google_service_account.api.name
role = "roles/iam.workloadIdentityUser"
member = "serviceAccount:${var.project}.svc.id.goog[rarelens/rarelens-api]"
}
resource "google_service_account_iam_member" "pipeline_workload_identity" {
count = var.deploy_kubernetes ? 1 : 0
service_account_id = google_service_account.pipeline.name
role = "roles/iam.workloadIdentityUser"
member = "serviceAccount:${var.project}.svc.id.goog[rarelens/rarelens-pipeline]"
}
resource "google_project_iam_member" "api_sql" {
project = var.project
role = "roles/cloudsql.client"
member = "serviceAccount:${google_service_account.api.email}"
}
resource "google_project_iam_member" "api_pubsub" {
project = var.project
role = "roles/pubsub.publisher"
member = "serviceAccount:${google_service_account.api.email}"
resource "google_pubsub_topic_iam_member" "api_publish" {
topic = google_pubsub_topic.vcf_uploaded.name
role = "roles/pubsub.publisher"
member = "serviceAccount:${google_service_account.api.email}"
}
resource "google_pubsub_subscription_iam_member" "pipeline_subscribe" {
subscription = google_pubsub_subscription.vcf_uploaded_argo.name
role = "roles/pubsub.subscriber"
member = local.pipeline_sa_member
}
resource "google_storage_bucket_iam_member" "pipeline_data" {
bucket = google_storage_bucket.data.name
role = "roles/storage.objectAdmin"
member = "serviceAccount:${google_service_account.pipeline.email}"
member = local.pipeline_sa_member
}
resource "google_artifact_registry_repository_iam_member" "pipeline_pull" {
repository = google_artifact_registry_repository.images.name
location = var.region
role = "roles/artifactregistry.reader"
member = local.pipeline_sa_member
}
# Nextflow's google-batch executor submits Batch jobs that run as the pipeline SA itself.
resource "google_project_iam_member" "pipeline_batch" {
for_each = toset(["roles/batch.jobsEditor", "roles/batch.agentReporter", "roles/logging.logWriter"])
project = var.project
role = each.value
member = local.pipeline_sa_member
}
resource "google_service_account_iam_member" "pipeline_act_as_self" {
service_account_id = google_service_account.pipeline.name
role = "roles/iam.serviceAccountUser"
member = local.pipeline_sa_member
}
infra/terraform/locals.tf
+16
View File
@@ -0,0 +1,16 @@
locals {
registry = "${var.region}-docker.pkg.dev/${var.project}/rarelens"
# join("", ...) rather than one(...): with count = 0 these collapse to "" instead of null.
db_credentials = "${join("", google_sql_user.api[*].name)}:${join("", random_password.pg[*].result)}"
db_name = join("", google_sql_database.rarelens[*].name)
db_private_ip = join("", google_sql_database_instance.pg[*].private_ip_address)
# The API reaches Cloud SQL through its cloud-sql-proxy sidecar on localhost; pipeline tasks
# (Google Batch VMs, Argo pods) use the private IP inside the VPC. With deploy_cloud_sql = false
# both use the URL you supplied, which is expected to be reachable over TLS.
api_database_url = var.deploy_cloud_sql ? "postgresql+asyncpg://${local.db_credentials}@127.0.0.1:5432/${local.db_name}" : var.database_url
pipeline_database_url = var.deploy_cloud_sql ? "postgresql://${local.db_credentials}@${local.db_private_ip}:5432/${local.db_name}" : var.database_url
pipeline_sa_member = "serviceAccount:${google_service_account.pipeline.email}"
}
infra/terraform/network.tf
+28 -1
View File
@@ -1,4 +1,31 @@
resource "google_compute_network" "vpc" {
name = "rarelens-vpc"
auto_create_subnetworks = true
auto_create_subnetworks = false
depends_on = [google_project_service.enabled]
}
# Shared by GKE and the Google Batch VMs that run pipeline tasks (pipeline/nextflow.config).
resource "google_compute_subnetwork" "gke" {
name = "rarelens-gke"
region = var.region
network = google_compute_network.vpc.id
ip_cidr_range = "10.10.0.0/20"
private_ip_google_access = true
}
# Private services access, so Cloud SQL gets a private IP inside this VPC. Only needed with it.
resource "google_compute_global_address" "private_services" {
count = var.deploy_cloud_sql ? 1 : 0
name = "rarelens-private-services"
purpose = "VPC_PEERING"
address_type = "INTERNAL"
prefix_length = 16
network = google_compute_network.vpc.id
}
resource "google_service_networking_connection" "private_services" {
count = var.deploy_cloud_sql ? 1 : 0
network = google_compute_network.vpc.id
service = "servicenetworking.googleapis.com"
reserved_peering_ranges = [google_compute_global_address.private_services[0].name]
}
infra/terraform/outputs.tf
+14 -5
View File
@@ -1,5 +1,14 @@
output "cluster_name" { value = google_container_cluster.rarelens.name }
output "sql_connection" { value = google_sql_database_instance.pg.connection_name }
output "data_bucket" { value = google_storage_bucket.data.name }
output "wif_provider" { value = google_iam_workload_identity_pool_provider.github.name }
output "ci_sa" { value = google_service_account.ci.email }
output "web_url" {
description = "The one URL to share"
value = google_cloud_run_v2_service.web.uri
}
output "api_url" { value = google_cloud_run_v2_service.api.uri }
output "nextflow_job" { value = google_cloud_run_v2_job.nextflow.name }
output "data_bucket" { value = google_storage_bucket.data.name }
output "cluster_name" { value = one(google_container_cluster.rarelens[*].name) }
output "sql_connection" { value = one(google_sql_database_instance.pg[*].connection_name) }
output "sql_private_ip" { value = one(google_sql_database_instance.pg[*].private_ip_address) }
output "wif_provider" { value = google_iam_workload_identity_pool_provider.github.name }
output "ci_sa" { value = google_service_account.ci.email }
output "api_sa" { value = google_service_account.api.email }
output "pipeline_sa" { value = google_service_account.pipeline.email }
infra/terraform/pubsub.tf
+11
View File
@@ -0,0 +1,11 @@
resource "google_pubsub_topic" "vcf_uploaded" {
name = "vcf-uploaded"
depends_on = [google_project_service.enabled]
}
# Consumed by the Argo Events EventSource in infra/argo-workflows/events.yaml.
resource "google_pubsub_subscription" "vcf_uploaded_argo" {
name = "vcf-uploaded-argo"
topic = google_pubsub_topic.vcf_uploaded.id
ack_deadline_seconds = 60
}
infra/terraform/secrets.tf
+41
View File
@@ -0,0 +1,41 @@
# Copied into Kubernetes secrets by `make gcp-secrets PROJECT=<id>`.
resource "google_secret_manager_secret" "api_database_url" {
secret_id = "rarelens-api-database-url"
replication {
auto {}
}
depends_on = [google_project_service.enabled]
}
resource "google_secret_manager_secret_version" "api_database_url" {
secret = google_secret_manager_secret.api_database_url.id
secret_data = local.api_database_url
lifecycle {
precondition {
condition = var.deploy_cloud_sql || var.database_url != ""
error_message = "Set database_url (e.g. a Neon URL), or deploy_cloud_sql = true."
}
}
}
# The id must match the Nextflow `secret = 'DATABASE_URL'` directive in pipeline/nextflow.config:
# on Google Batch, Nextflow resolves secrets from Secret Manager by name.
resource "google_secret_manager_secret" "pipeline_database_url" {
secret_id = "DATABASE_URL"
replication {
auto {}
}
depends_on = [google_project_service.enabled]
}
resource "google_secret_manager_secret_version" "pipeline_database_url" {
secret = google_secret_manager_secret.pipeline_database_url.id
secret_data = local.pipeline_database_url
}
resource "google_secret_manager_secret_iam_member" "pipeline_database_url" {
secret_id = google_secret_manager_secret.pipeline_database_url.id
role = "roles/secretmanager.secretAccessor"
member = local.pipeline_sa_member
}
infra/terraform/storage.tf
+3 -2
View File
@@ -2,6 +2,8 @@ resource "google_storage_bucket" "data" {
name = "${var.project}-rarelens-data"
location = var.region
uniform_bucket_level_access = true
public_access_prevention = "enforced"
depends_on = [google_project_service.enabled]
lifecycle_rule {
condition {
age = 30
@@ -15,6 +17,5 @@ resource "google_artifact_registry_repository" "images" {
repository_id = "rarelens"
location = var.region
format = "DOCKER"
depends_on = [google_project_service.enabled]
}
resource "google_pubsub_topic" "vcf_uploaded" { name = "vcf-uploaded" }
infra/terraform/variables.tf
+55 -5
View File
@@ -1,9 +1,59 @@
variable "project" { type = string }
variable "project" {
description = "GCP project id"
type = string
}
variable "region" {
type = string
default = "europe-west2" # London: keeps public genomic test data and the Cambridge team in one jurisdiction
description = "GCP region for every regional resource"
type = string
default = "europe-west2" # London: keeps public genomic test data and the Cambridge team in one jurisdiction
}
variable "github_repo" {
type = string
default = "lynchaos/rarelens"
description = "owner/name of the GitHub repo allowed to push images via Workload Identity Federation"
type = string
default = "lynchaos/rarelens"
}
variable "deletion_protection" {
description = "Protect the GKE cluster and Cloud SQL instance from `terraform destroy`; set false to tear the lab down"
type = bool
default = true
}
variable "deploy_kubernetes" {
description = "Create the GKE cluster (Argo/ArgoCD track). Off by default: it costs ~$150/month idle, while the serverless track costs ~£1 (docs/cloud.md)"
type = bool
default = false
}
variable "deploy_cloud_sql" {
description = "Create a Cloud SQL instance (~$10/month). Off by default: set database_url to a free scale-to-zero Postgres such as Neon"
type = bool
default = false
}
variable "database_url" {
description = "Postgres URL used when deploy_cloud_sql is false, e.g. postgresql+asyncpg://user:pass@host/db?sslmode=require"
type = string
default = ""
sensitive = true
}
variable "image_tag" {
description = "Image tag deployed to Cloud Run; CI pushes the commit SHA"
type = string
default = "latest"
}
variable "model_uri" {
description = "Optional model artifact to score with, e.g. gs://<project>-rarelens-data/models/pathogenicity/1. Empty means use the MLflow registry"
type = string
default = ""
}
variable "max_instances" {
description = "Cloud Run instance ceiling per service: scale-to-zero bounds the floor, this bounds the bill"
type = number
default = 2
}
infra/terraform/versions.tf
+4 -1
View File
@@ -4,7 +4,10 @@ terraform {
google = { source = "hashicorp/google", version = "~> 6.0" }
random = { source = "hashicorp/random", version = "~> 3.6" }
}
backend "gcs" { bucket = "REPLACE-tfstate", prefix = "rarelens" }
# Partial config: terraform init -backend-config="bucket=<your-tfstate-bucket>"
backend "gcs" {
prefix = "rarelens"
}
}
provider "google" {