Why architecture matters here
Tracing without a plan produces cost surprises: 100% sampling on high-traffic services burns terabytes. Tracing without governance produces useless traces — spans without attributes, or with too many. Tracing without correlation makes you switch between three tools per incident.
The architecture matters because these decisions compound. Head sampling means you decide at request start; tail sampling means you decide after you see the whole trace. Cardinality budgets on attributes protect the backend. Correlation via trace ID makes logs and metrics useful.
With a plan, tracing becomes the fastest path to root cause for any distributed incident.
The architecture: every piece explained
The top strip is the instrumentation. Service A starts a span when a request arrives. Context propagation — W3C traceparent header — carries the trace ID and parent span ID to downstream calls. Service B / C create child spans linked to the parent. OTel SDK collects spans and hands them to the exporter.
The middle row is the pipeline. Head sampling makes the decision at ingress based on trace ID hash — cheap but decides before the request outcome is known. Tail sampling buffers the whole trace and decides after — expensive but can keep 100% of errors + a percentage of successes. OTel Collector processes, filters, transforms, and routes to backends. Storage is Jaeger, Tempo, Honeycomb, or a hosted option — with retention policies.
The lower rows are usage and governance. Query + UI supports search by attribute, waterfall view, and error highlighting. Correlation uses trace ID as the join key with logs (via structured logs) and metrics (exemplars). Governance sets attribute conventions, cardinality budgets, and PII scrubbing rules.
End-to-end flow
End-to-end: a user hits an endpoint. Service A starts a trace with a new trace ID and a root span. Head sampling includes this trace based on the trace ID modulo. Service A calls Service B and C in parallel; the OTel SDK injects traceparent headers. B and C create child spans, do work, and return. All spans flow to the Collector, which strips PII from HTTP body attributes and routes to Tempo. A p95 latency alert fires 10 minutes later; the SRE opens Grafana, follows the exemplar from the metric to the trace, and sees Service C waiting 2 seconds on a downstream. Root cause found in one hop. Logs correlated by trace ID confirm the specific query that was slow.