Skip to content

IaC · observe

Prototype to production

Rebuild it from the repo

An AI feature on one VM, secrets in an env file, deploys by SSH. Moved to containers, a pipeline, more than one instance, infrastructure as code, and alerts on error rate, latency, and queue depth.

  • Docker
  • Kubernetes
  • Terraform
  • GitHub Actions
  • Prometheus

PIPELINE

Build, test, push image, deploy

RUNTIME

Load-balanced API and workers

SECRETS

Injected at runtime, not in the image

ALERTS

Error rate, latency, queue depth

01The problem

The feature ran as one process on one VM. A crash was an outage. A deploy was an outage. Secrets and config sat in a file on the box. The only person who could repeat a deploy was the person who had done it last time, from memory.

Monitoring stopped at whether the process was still running. The team heard about a slow or failing service when a user said so.

02What we designed

The API and its background workers are containers. A pipeline builds, tests, pushes an image, and deploys it. More than one API instance sits behind a load balancer, so one dead process is not the service.

Those two workloads scale on their own. A single-service platform would have tied their replica counts together, which is why this runs on Kubernetes. That split is the reason. A short MVP does not get a cluster for its own sake.

The environment is Terraform. Rebuilding it means applying the repo, not reconstructing a console session. Secrets are injected at runtime from a managed store. They are not in the image and not in the repository.

Alerts fire on error rate, latency, and queue depth.

03What I owned

The pipeline, the split between API instances and workers, the Terraform layout, and which three signals page someone.