Go · ingest
Telemetry ingestion
Shed load before it queues
Python ingestion needed more instances than the throughput justified. GC pauses under load dropped client connections, retries added load, and the fleet was sized for that loop. The ingest tier is now a Go worker pool.
- Go
- gRPC
- Prometheus
- Kubernetes
POOL
Goroutines over a bounded channel
DEDUP
In-memory LRU with a short TTL
FAN-OUT
Async gRPC to stream and cold store
SHED
429 once the queue hits its bound
01The problem
The ingest path was Python. It worked, and it needed more instances than the throughput justified. Under load, GC pauses dropped client connections. Clients retried. Retries landed while the process was already behind, so the fleet was sized for that loop, not for the events.
Each event is small: validate, drop a duplicate, then hand it to a stream processor and a cold-storage sink. Duplicates only mattered inside a short window. The latency budget did not have room for another network hop on that check.
02What we designed
A Go service. A small number of goroutines per core pull from an in-memory channel queue, so throughput tracks cores instead of instance count.
Dedup is a bounded LRU with a short TTL, in the process. A restart can let a duplicate through. That was the trade for keeping the hot path local.
Fan-out runs on two goroutine pools, over gRPC, so a slow sink does not block accept. REST stays at the public edge. Once the queue hits its bound, the service returns 429 instead of letting the queue grow into an outage.
03What I owned
The worker-pool shape, the in-memory dedup bound, the split between the REST edge and gRPC fan-out, and the load-shed contract.