IaC · observe
Prototype to production
An AI feature on one VM, secrets in an env file, deploys by SSH. Moved to containers, a pipeline, more than one instance, infrastructure as code, and alerts on error rate, latency, and queue depth.
PIPELINE, CLUSTER, TRUST
01Principles
- Rebuild from the repo. If it is not in Terraform or the pipeline, it is not part of the environment.
- Two workloads, two scales. The API and the background workers do not share a replica count.
- Secrets stay out of the artifact. A leaked image or a leaked repo is not a leaked credential.
- Few alerts. Error rate, latency, queue depth. The rest can live on a dashboard.
02System map
Rendering diagram…
03What changed when something breaks
One instance dying is a restart. The load balancer stops sending it traffic. The other instances keep the service up.
A bad deploy rolls back through the same pipeline. The previous path was SSH and a process restart, which only the last person on the box could repeat.
A leaked image does not contain the database password or the API key. Those are read at runtime.
An alert means error rate, latency, or queue depth moved. A process that is up and useless used to look healthy.