Kubernetes and GitOps

Off by default. terraform apply -var deploy_kubernetes=true turns it on; it exists to demonstrate the GitOps path and is meant to be destroyed afterwards.

Layout

infra/k8s/base/              api, web, postgres, ingress
infra/k8s/overlays/local/    kind: in-cluster Postgres, NodePort
infra/k8s/overlays/gcp/      Cloud SQL, Workload Identity, image tags
infra/argo-workflows/        annotate WorkflowTemplate, EventSource, Sensor, RBAC
infra/argocd/app.yaml        the Application that reconciles the cluster

Locally

make kind      # builds the images, loads them into a kind cluster, applies the local overlay

The GitOps loop

A pull request merges to main.
CI builds and pushes images to Artifact Registry.
CI bumps the image tags in infra/k8s/overlays/gcp.
ArgoCD sees the commit and reconciles the cluster.

Deployment is therefore a commit, and rollback is a revert.

Argo Workflows and Argo Events

annotate.yaml is a WorkflowTemplate running the same containers the Nextflow pipeline uses. events.yaml holds the Pub/Sub EventSource and the Sensor that triggers the template when the API publishes a job. rbac.yaml grants the sensor permission to create workflows in the rarelens namespace and nothing else.

The ConfigMap trap

Kustomize appends a content hash to generated ConfigMap names so that a config change forces a rollout. That substitution only happens for workloads Kustomize considers in scope. Both overlays were missing namespace: rarelens, so the hashed name was generated but never substituted, and pods mounted a ConfigMap name that no longer existed.

The symptom is a pod stuck in CreateContainerConfigError while every manifest reads correctly. See Gotchas.

rarelens

Understanding it

Working on it

Running it

When it goes wrong