Skip to content

Go · ingest

Telemetry ingestion

Shed load before it queues

Python ingestion needed more instances than the throughput justified. GC pauses under load dropped client connections, retries added load, and the fleet was sized for that loop. The ingest tier is now a Go worker pool.

  • Go
  • gRPC
  • Prometheus
  • Kubernetes

POOL

Goroutines over a bounded channel

DEDUP

In-memory LRU with a short TTL

FAN-OUT

Async gRPC to stream and cold store

SHED

429 once the queue hits its bound

01The problem

The ingest path was Python. It worked, and it needed more instances than the throughput justified. Under load, GC pauses dropped client connections. Clients retried. Retries landed while the process was already behind, so the fleet was sized for that loop, not for the events.

Each event is small: validate, drop a duplicate, then hand it to a stream processor and a cold-storage sink. Duplicates only mattered inside a short window. The latency budget did not have room for another network hop on that check.

02What we designed

A Go service. A small number of goroutines per core pull from an in-memory channel queue, so throughput tracks cores instead of instance count.

Dedup is a bounded LRU with a short TTL, in the process. A restart can let a duplicate through. That was the trade for keeping the hot path local.

Fan-out runs on two goroutine pools, over gRPC, so a slow sink does not block accept. REST stays at the public edge. Once the queue hits its bound, the service returns 429 instead of letting the queue grow into an outage.

03What I owned

The worker-pool shape, the in-memory dedup bound, the split between the REST edge and gRPC fan-out, and the load-shed contract.