Why architecture matters here
The economics are stark. A mid-sized platform emitting 500k spans/second produces tens of terabytes of trace data per day; storing it all costs more than the databases it monitors, yet the value is concentrated in a sliver — the 0.1% of traces that are broken, slow, or rare. Uniform sampling keeps a representative sample of boring traffic, which is precisely backwards: nobody queries tracing to admire successful 40ms requests. The architecture matters because it is the mechanism that lets you keep 100% of the interesting sliver while paying storage for ~1–5% of total volume.
The hard requirement that shapes everything is trace completeness at decision time. A policy like 'keep if any span errored' is only correct if the evaluator sees every span of the trace; if spans for one trace scatter across ten collector replicas, each replica sees a fragment and the verdicts are garbage — one replica keeps its half, another drops the other half, and the backend fills with torn traces that erode trust in the whole system. So routing must be sticky by trace ID, which turns a stateless fleet into a consistent-hashed one and imports all the classic problems: rebalancing on scale-out, hot shards from trace-volume skew, and ownership churn during deploys.
The second shaping constraint is that sampling bias must not poison metrics. Once tail sampling runs, trace counts are useless for rates and the kept-trace latency distribution is deliberately skewed toward the slow tail. Teams that compute SLOs from sampled traces silently report error rates inflated by 20× (errors are kept, successes are dropped). The architecture therefore always pairs the sampler with a span-metrics pipeline that computes request/error/duration aggregates from the full stream before the drop decision — traces become the drill-down, never the denominator.
The architecture: every piece explained
Top row: getting every span to one owner. Instrumented services run with sampling effectively disabled — the SDK exports 100% of spans (an AlwaysOn sampler or a high head ratio as a safety valve under extreme load). Sidecar or node agent collectors batch spans, attach resource attributes (service, version, zone), and forward OTLP. The trace-ID load balancer is the linchpin: a thin collector layer running a load-balancing exporter that consistent-hashes each span's trace ID over the downstream fleet, so every span of trace abc123 lands on the same sampling collector regardless of which of fifty services emitted it. Fleet membership comes from DNS or the orchestrator's endpoint list; the hash ring must be identical on every balancer replica or ownership tears.
Middle row: the decision machinery inside each sampling collector. The decision buffer holds arriving spans grouped by trace ID for the decision window — typically 10–30 seconds after the first span — long enough for stragglers (async work, retries, slow leaf calls) to arrive. Memory is the budget here: spans/second × window × average span size, per replica; the buffer is bounded, with back-pressure and an emergency degrade path (fall back to probabilistic sampling) when full. When a trace's window closes, the policy evaluator runs the ordered policy set: keep if any span has error status; keep if root duration exceeds a threshold (often per-route thresholds, since a checkout at 2s and a health check at 2s are different events); keep rare attributes (new route, new version, specific tenant under investigation); then a ratio policy keeps a small percentage of the remainder as an unbiased baseline. Policies compose as OR — first match keeps.
The decision cache records each verdict for minutes afterward, because spans can arrive after the window closes — a retry queued behind a slow consumer, a batch-processed async hop. Late spans for a kept trace pass straight through (the backend merges them); late spans for a dropped trace are dropped consistently, avoiding orphaned fragments. Without this cache, every late span becomes a torn trace.
Bottom row: the two outputs. Kept traces flow to the trace backend at full fidelity. In parallel — and crucially, upstream of the drop — the span-metrics processor aggregates every span into RED metrics (rate, errors, duration histograms) by service and route, exported to the metrics backend with exemplar links pointing at kept trace IDs, so a latency spike on a dashboard clicks through to a kept trace that exhibits it. Metrics answer 'how much, how often'; sampled traces answer 'what exactly happened' — the architecture keeps each honest at its job.
End-to-end flow
Follow one request. Step 1 — emission. A checkout request enters the edge gateway, which starts trace abc123; over 800ms it fans through nine services and forty-one spans, one of which — an inventory call — retries twice and errors once before succeeding. Every SDK exports every span to its local agent; nothing is dropped at the head. Step 2 — sticky routing. Agents forward to the balancer tier; each balancer hashes abc123 to sampling collector #7. All forty-one spans, from nine services across three zones, converge on #7 over the next second.
Step 3 — buffering and counting. Collector #7 opens a buffer entry at the first span's arrival and starts the 15-second window clock. Simultaneously the span-metrics processor increments request counts and duration histograms for all nine services — the metrics path has already recorded this trace's existence regardless of the sampling verdict. Step 4 — decision. At window close, the evaluator walks the policies: the error policy matches (the failed retry span has error status), so the whole trace is kept — verdict cached, all forty-one spans exported to the trace backend. Had the trace been clean and fast, only the 2% ratio policy could have saved it; the other 98% of clean traces evaporate here, which is the entire cost model working as intended.
Step 5 — the straggler. Four seconds after the verdict, a fulfillment-worker span for abc123 arrives — it was sitting in a queue. The decision cache says 'kept', so it passes through and the backend stitches it into the stored trace. Step 6 — consumption. An hour later, an on-call engineer sees the inventory error-rate panel tick up. The panel is span-metrics data — it counted every request, so the rate is true. She clicks the exemplar on the spike and lands in trace abc123, complete with the retry storm and the queued fulfillment span, because the one trace she needed is one the sampler was designed to never lose. The system kept perhaps 3% of the hour's spans and 100% of the hour's evidence.