Skip to content

IaC · observe

Prototype to production

An AI feature on one VM, secrets in an env file, deploys by SSH. Moved to containers, a pipeline, more than one instance, infrastructure as code, and alerts on error rate, latency, and queue depth.

IAC LAYOUT, SECRETS, ALERTS

01Pipeline

StageWhat it does
BuildOne image for the API, one for the workers
TestThe checks that used to be "it started after I SSH'd in"
PushBoth images, tagged from the commit
DeployRoll the new images. Rollback is the same path with the previous tag

The deploy is not a shell history on the VM.

02What lives where

Terraform owns the environment: the cluster, the load balancer, the managed database, the secrets store, and the identities the workloads assume. The application repo owns the images, the pipeline, and the alert rules that read application metrics.

A console edit is drift. The next apply is supposed to put it back.

ConcernWhere it lives
Cluster, network, database, secrets storeTerraform
Images, deploy, rollbackApplication pipeline
Error rate, latency, queue depthAlert rules on metrics the app already emits

03Secrets

Nothing secret is baked into the image or committed next to the code. The process reads credentials at start from the managed store. Config that is not secret can ship with the deploy. Credentials cannot.

04Alerts

Whether the process is running stays as a liveness probe. It does not page anyone.

SignalWhat it catches
Error rateFailures a process check never saw
LatencyA service that is up and slow
Queue depthWorkers falling behind while the API still returns 200

Kubernetes is here because the API and the workers need different replica counts. A single-service deploy would have made that split awkward.