Skip to content

IaC · observe

Prototype to production

An AI feature on one VM, secrets in an env file, deploys by SSH. Moved to containers, a pipeline, more than one instance, infrastructure as code, and alerts on error rate, latency, and queue depth.

PIPELINE, CLUSTER, TRUST

01Principles

  • Rebuild from the repo. If it is not in Terraform or the pipeline, it is not part of the environment.
  • Two workloads, two scales. The API and the background workers do not share a replica count.
  • Secrets stay out of the artifact. A leaked image or a leaked repo is not a leaked credential.
  • Few alerts. Error rate, latency, queue depth. The rest can live on a dashboard.

02System map

SYSTEM MAP

Rendering diagram…

03What changed when something breaks

One instance dying is a restart. The load balancer stops sending it traffic. The other instances keep the service up.

A bad deploy rolls back through the same pipeline. The previous path was SSH and a process restart, which only the last person on the box could repeat.

A leaked image does not contain the database password or the API key. Those are read at runtime.

An alert means error rate, latency, or queue depth moved. A process that is up and useless used to look healthy.