Go · ingest
Telemetry ingestion
Python ingestion needed more instances than the throughput justified. GC pauses under load dropped client connections, retries added load, and the fleet was sized for that loop. The ingest tier is now a Go worker pool.
POOL, DEDUP, GRPC, METRICS
01Accept path
Public clients speak HTTP. The handler validates the event, then tries a non-blocking send on a bounded channel. A full channel is a 429. The handler does not wait for fan-out.
select {
case queue <- event:
// accepted
default:
http.Error(w, "shed", http.StatusTooManyRequests)
}
Workers are a fixed set of goroutines, sized to cores, reading that channel. Throughput moves with cores. Instance count is what you add when those cores are already busy.
02Dedup
A bounded LRU with a short TTL, in the process. An event seen inside the window is dropped. An event seen after the TTL, or after a restart, is treated as new.
A shared store is not on this path. Duplicates only mattered inside that window, and a lookup on every accept would put a network hop in front of the budget.
03Fan-out
Two goroutine pools, one per sink. Each pool takes accepted events and sends them over gRPC. The pools sit off the accept handler, so a slow write on one sink does not block the other, and neither blocks HTTP until the queue itself is full.
gRPC is internal. The schema is shared by the ingest service and the two sinks. REST stays where external clients already are.
| Edge | Protocol | Blocks accept? |
|---|---|---|
| Public ingest | HTTP | Only when the queue is full, and then it sheds |
| Stream processor | gRPC | No |
| Cold-storage sink | gRPC | No |
04Metrics
Prometheus, on the two signals that explain a shed:
| Signal | Why it is there |
|---|---|
| Queue depth | Shows the bound filling before clients see 429 |
| Shed count | Separates "we refused" from "we timed out" |
The old failure was a pause, a dropped connection, and a retry storm that looked like more traffic. Queue depth and shed count are what you read instead.