IaC · observe
Prototype to production
Rebuild it from the repo
An AI feature on one VM, secrets in an env file, deploys by SSH. Moved to containers, a pipeline, more than one instance, infrastructure as code, and alerts on error rate, latency, and queue depth.
- Docker
- Kubernetes
- Terraform
- GitHub Actions
- Prometheus
PIPELINE
Build, test, push image, deploy
RUNTIME
Load-balanced API and workers
SECRETS
Injected at runtime, not in the image
ALERTS
Error rate, latency, queue depth
01The problem
The feature ran as one process on one VM. A crash was an outage. A deploy was an outage. Secrets and config sat in a file on the box. The only person who could repeat a deploy was the person who had done it last time, from memory.
Monitoring stopped at whether the process was still running. The team heard about a slow or failing service when a user said so.
02What we designed
The API and its background workers are containers. A pipeline builds, tests, pushes an image, and deploys it. More than one API instance sits behind a load balancer, so one dead process is not the service.
Those two workloads scale on their own. A single-service platform would have tied their replica counts together, which is why this runs on Kubernetes. That split is the reason. A short MVP does not get a cluster for its own sake.
The environment is Terraform. Rebuilding it means applying the repo, not reconstructing a console session. Secrets are injected at runtime from a managed store. They are not in the image and not in the repository.
Alerts fire on error rate, latency, and queue depth.
03What I owned
The pipeline, the split between API instances and workers, the Terraform layout, and which three signals page someone.