Why architecture matters here
The reason fixed-rate sampling needs replacing is that trace value is wildly non-uniform while trace volume is too, and the two are often inversely correlated. The operations that produce the most spans — health checks, cache hits, static asset serves — are frequently the least interesting, while the operations you most want to see when debugging — a rare payment failure, a slow-path fallback, a specific tenant's request — produce few spans. A single global rate cannot serve both: set it high enough to capture the rare operation and you drown in (and pay for) the routine firehose; set it low enough to afford the firehose and you sample the rare operation into invisibility.
The consequence is not just cost, it is blindness precisely where it hurts. An engineer paged for elevated errors on a low-traffic endpoint opens the tracing UI and finds no example traces, because 1% of a trickle is nothing. The tool fails at the exact moment it is needed. Meanwhile the backend is straining under millions of health-check traces nobody will ever look at. Fixed-rate sampling optimizes the wrong thing: it makes the sample proportional to volume when what you want is a sample proportional to investigative value, floored so that even rare operations are represented.
Adaptive sampling reframes the problem as budget allocation. There is a total volume the backend can afford — a spans-per-second or traces-per-second budget — and the question is how to spend it. Instead of spending it proportionally to traffic (which is what a fixed rate does), the governor spends it to maximize coverage: it guarantees each operation a minimum representation, then distributes the remaining budget so that no single high-volume operation can consume it all. A rare operation gets sampled at or near 100%; a firehose operation gets sampled at whatever tiny rate keeps it within its share. The budget is honored, and every operation is visible.
The priority dimension addresses a different failure of uniform sampling: it is blind to outcome. Whether a request succeeded in 5ms or failed after a 5-second timeout, a probabilistic sampler keeps it with the same probability, so the interesting tail is sampled as sparsely as the boring bulk. But the interesting tail is the entire point of tracing. Adaptive systems therefore layer outcome-aware rules on top of the volume-based rates — always keep errors, always keep traces over a latency threshold, always keep explicitly-flagged debug traces — so the anomalous is captured deterministically while the normal is captured statistically.
Architecturally, the key insight is that sampling must become a closed-loop control system rather than a static configuration. Traffic is not constant: it surges at peak hours, shifts as features launch, and spikes during incidents — which is exactly when a fixed rate is most wrong, because an incident both increases volume and increases the value of each trace. A governor that measures actual throughput and adjusts rates every interval tracks these changes automatically, holding the budget steady as traffic doubles by halving the rates, and reallocating toward newly-hot operations. Treating sampling as a feedback loop with a target, a measurement, and an actuator is what lets a tracing system stay both affordable and useful as the workload underneath it moves.
The architecture: every piece explained
Trace the components. The spans and traces are the firehose: the raw telemetry every instrumented service emits. The sampler sits in the emission path (in the SDK, an agent, or a collector) and makes the keep/drop decision for each trace using a rate table — a set of per-key sampling probabilities, where a key is typically an operation-and-service pair (for example checkout-service:POST /pay). The sampler is deliberately simple and fast; the intelligence lives in the governor that produces the table.
The rate governor is the controller. It consumes measurements of actual trace throughput — how many traces each key produced in the last interval, held in per-key counters — and compares the totals against the configured target budget. The budget allocator then computes new probabilities: it gives each key a fair share of the budget, sampling low-volume keys at up to 100% and high-volume keys down to whatever rate keeps them within their allocation. The allocation policy is where fairness lives — a common approach guarantees every key a floor (so rare operations are never starved to zero) and then divides the surplus, preventing one firehose key from crowding out the rest.
The priority rules operate alongside the volume-based rates. Certain traces are force-kept regardless of their key's computed probability: any trace containing an error span, any trace whose duration exceeds a latency threshold, any trace explicitly marked for debugging. These decisions are often made at the tail (after the trace completes, so the error or latency is known), which is why adaptive sampling frequently pairs with tail-based sampling — the full trace is buffered briefly so the sampler can see the outcome before deciding. Head-based adaptive sampling, by contrast, decides at the root span using only the rate table, trading outcome-awareness for lower buffering cost.
The rate table push is the actuator: the governor distributes the freshly computed probabilities back to every sampler, so the next interval's decisions reflect current traffic. This distribution must be efficient and eventually consistent — samplers operate on the last table they received, and a brief lag between measurement and new rates is acceptable because the loop self-corrects. Each sampling decision records the probability at which the trace was kept, written into the trace's metadata.
That recorded probability is the piece that makes uneven sampling statistically honest, and it is easy to underrate. If you keep a firehose operation at 0.1% and a rare one at 100%, then naively counting sampled traces would make the firehose look a thousand times smaller relative to reality than it is. Because each trace carries the inverse of its keep-probability as a weight, downstream aggregation can reconstruct accurate totals — a trace kept at 0.1% counts for 1,000, one kept at 100% counts for 1 — so dashboards and rate calculations remain correct despite the adaptive, non-uniform sampling. Without this weighting, adaptive sampling would buy affordability at the cost of lying about volume; with it, you get both economy and accurate counts, which is the whole point of the design.
End-to-end flow
Walk a normal interval. The backend budget is 10,000 traces/second. In the last interval, health-check:GET /healthz produced 8,000 traces/s, catalog:GET /items produced 1,500/s, and checkout:POST /pay produced 20/s. A fixed 1% rate would keep 80, 15, and 0.2 traces/s respectively — the checkout endpoint effectively invisible. The governor instead allocates: it keeps checkout at 100% (20/s, trivially within budget), catalog generously, and samples the health-check firehose down hard so the total lands near the 10,000 budget. Rare and interesting traffic is now fully represented; the firehose is throttled to its fair share.
Now a priority override. During that interval, three checkout requests errored and one catalog request took 4 seconds. Those four traces are force-kept by the priority rules regardless of their key's rate — the errors because they are errors, the slow one because it crossed the latency threshold. So even if catalog's computed rate were modest, the anomalous catalog trace is captured. An engineer investigating the error or the slow request finds concrete examples, which is the entire reason tracing exists.
Now a traffic shift. A marketing push triples catalog traffic to 4,500/s mid-day. The governor's next measurement sees the surge; the budget allocator recomputes, lowering catalog's sampling probability so its contribution stays within its share of the 10,000 budget, and correspondingly adjusting the others. The total stays near budget without human intervention — the loop absorbed a 3x traffic change by moving rates, not by blowing the budget or dropping to a panic rate. When the push ends and traffic subsides, the governor raises catalog's rate back up, restoring fuller coverage.
Consider an incident spike. A dependency fails and checkout starts erroring on 30% of requests; its volume also jumps as clients retry. This is the worst case for fixed-rate sampling — more volume and more value at once. Adaptive sampling handles both axes: the priority rule force-keeps every error trace, so the investigation has abundant examples of the failure; and the governor keeps the non-error checkout traffic within budget so the surge does not overwhelm the backend. The engineer gets maximum visibility into the failure exactly when it matters, and the tracing pipeline stays up because the budget held.
Finally, count reconstruction during analysis. A dashboard wants the true request rate for the health-check endpoint, which was sampled at, say, 0.12% during the incident. It does not count kept traces directly; it sums each kept trace's weight (the inverse of its recorded keep-probability), so 96 kept traces at 0.12% reconstruct to roughly 80,000 — the real rate. Because every decision recorded its probability, the non-uniform sampling does not distort the numbers: the expensive firehose was barely stored yet its volume is reported accurately, and the rare endpoint was fully stored and reported accurately too. Economy and correctness coexist because the weight travels with the trace.