Why architecture matters here
The architecture matters because the alternative to sampling is not 'keep everything' — it is 'go bankrupt or fall over'. Trace data volume scales with request rate times fan-out times span count, which for a microservice estate is astronomical, and the cost lands three times over: CPU and memory to generate spans, network to export them, and storage plus indexing to retain and query them. Head sampling attacks all three at the earliest possible point. Because the drop decision is made before the child spans exist, an unsampled trace never generates them, never serializes them, never ships them — the savings are near-total, not just a storage discount. That is the crucial contrast with tail-based sampling, which must first collect every span of a trace (paying generation and transport) and only then decide, buying better selectivity at much higher cost.
The second reason the design is subtle is consistency. A distributed trace spans many services, each of which could in principle make its own sampling choice. If service A keeps a span but service B independently drops the child, you get a broken trace — a root with missing branches — which is often worse than no trace at all because it misleads. Head sampling solves this by making the decision exactly once, at the root, and propagating it: the sampled bit travels in the trace context (the W3C traceparent header's flags), and every downstream service simply obeys it rather than re-deciding. When a service must make an independent decision (for a trace that entered without a decision), deterministic hashing of the shared trace-id ensures every service computes the same keep/drop, preserving whole traces without coordination.
The third architectural tension is statistical honesty. If you keep 1% of traces and then count 'requests per second' from your trace store, you will undercount by 100x — unless each kept trace carries the information needed to weight it back up. Head sampling's rate is knowable in advance (unlike tail sampling's), which makes unbiased extrapolation possible, but only if the system records the sampling probability so downstream analytics can correct for it. Teams that skip this quietly ship dashboards that are wrong by the sampling factor, and the architecture matters precisely because getting these three things right — cost, consistency, and correctability — is what separates useful cheap tracing from cheap misleading tracing.
The architecture: every piece explained
Top row: where and how the decision is made. The root span is the first span of a trace, created at the entry point — the edge service or gateway that received the request without an incoming trace context. Here the sampler runs: it might be a fixed probability (keep 1%), a rate limiter (keep at most N traces/sec), or a rule-based policy (always keep this route, sample that one lower). Whatever it decides becomes the sampled flag — a single bit in the trace context indicating keep or drop. Propagation carries that context, including the flag and the trace-id, to every downstream call via headers (W3C traceparent/tracestate), so the decision travels with the request.
Middle row: how downstream services stay consistent and how kept spans reach storage. Child services read the incoming trace context and honor the inherited bit — they do not re-sample; if the flag says keep, they record and export their spans, and if it says drop, they skip the cost entirely. For the case where a service is itself an entry point or the context is absent, consistent hashing of the trace-id (comparing a hash of the id against a threshold derived from the rate) guarantees that any service computing the decision independently arrives at the same answer, because they all hash the same id. The exporter emits only sampled spans to the backend, which stores and indexes them for query. Unsampled spans never take this path.
Bottom rows: making the policy smart. Adaptive rate control adjusts the sampling probability to hold a target throughput — say, a steady 100 traces/sec per service regardless of whether traffic is 1k or 100k rps — so cost stays bounded and low-traffic services aren't sampled into invisibility. Force-keep overrides are the escape hatch for head sampling's blindness: certain conditions known at the head — a debug flag on the request, a specific high-value customer, a critical route — can force the sampled bit on regardless of the base rate. (Errors and latency, known only later, are the domain of tail sampling, which head sampling complements rather than replaces.) The ops strip names the numbers that keep this honest: the effective sampling rate per service, coverage gaps, whether metrics are being biased by naive counting, and cost per service.
End-to-end flow
Trace one request through a head-sampled system. A user hits an API gateway; there is no incoming trace context, so this is a root. The gateway's sampler is configured for an adaptive target of 50 traces/sec; at the current load that works out to roughly a 2% probability. It rolls the dice — or rather, it derives the decision deterministically from the freshly-minted trace-id — and this one lands in the kept 2%. The sampled flag is set to keep, and the gateway writes a traceparent header carrying the trace-id and that flag onto its call to the downstream order service.
The order service receives the request, reads the trace context, and sees the keep bit. It does not run its own sampler; it simply records its span and propagates the same context to the three services it calls — inventory, pricing, and payment. Each of those, in turn, honors the inherited keep bit and records its spans. The result is a complete trace: root plus every branch, because a single decision governed all of them. Every span was exported to the backend, where the trace is now queryable end to end with no missing children.
Contrast the dropped 98%. For those requests, the gateway set the flag to drop, and every downstream service, reading that bit, skipped span recording and export entirely. No child spans were generated, nothing was serialized, nothing shipped — the trace cost essentially nothing beyond propagating a header. This is the efficiency that head sampling exists to deliver, and it is why a fleet can afford to instrument everything while storing a small, consistent fraction.
Now the wrinkle head sampling has to manage. Suppose one of the dropped requests actually errored — exactly the trace an operator would have wanted. Head sampling made its choice before the error existed, so absent an override, that trace is gone. Two mitigations play out in practice. First, a force-keep override: the client set a debug header, or the route was flagged critical, so the gateway forced the keep bit on regardless of the 2% base rate, and this error was captured because the condition was known at the head even though the error was not. Second, the honest answer for error-triggered capture is tail-based sampling layered alongside — a collector buffers spans and keeps traces that errored or ran slow — so the system uses head sampling for cheap broad coverage and tail sampling for the rare-but-important. Finally, the statistical correction: because the gateway recorded the sampling probability, the analytics layer multiplies each kept trace by its inverse (about 50x here) and reconstructs an unbiased request rate from the 2% it retained — the dashboards read true despite keeping almost nothing.