Skip to content

Go · ingest

Telemetry ingestion

Python ingestion needed more instances than the throughput justified. GC pauses under load dropped client connections, retries added load, and the fleet was sized for that loop. The ingest tier is now a Go worker pool.

HOT PATH, FAN-OUT, LOAD SHED

01Principles

  • Cores, not instances. A worker pool over a channel queue. Adding a machine is the last knob.
  • Hot path stays local. Dedup is memory. A shared store would buy a stronger guarantee and add a hop the budget did not have.
  • Accept and fan-out are different jobs. A slow downstream write must not stall ingestion.
  • Shed, don't buffer. A full queue is a 429, not an unbounded backlog.

02System map

SYSTEM MAP

Rendering diagram…

03What fails on purpose

A restart drops the in-memory dedup window. A duplicate that arrives in that gap can pass. The window is short, and the alternative was a network round trip on every event.

If the queue is full, the edge returns 429. Clients retry later. An unbounded queue would turn the same spike into timeouts, disconnects, and a second wave of retries.

The two sinks have their own pools. Backpressure on cold storage does not stop the stream processor, and neither stops the accept path until the queue itself is full.