Why architecture matters here
The architectural case for span metrics starts with the cost of the alternative. Maintaining separate metric instrumentation alongside tracing means two code paths to keep in sync: when someone adds an endpoint, they must remember to add both a span and a counter, and when the definitions diverge — the metric counts something the trace scopes differently — you get the worst kind of observability bug, where the dashboard and the traces disagree and nobody knows which to trust. Deriving metrics from spans makes the trace the single source of truth: the latency histogram is by construction the distribution of the same span durations you can drill into, so the graph and the trace can never contradict each other. That coherence is not a convenience; it is what makes the 'metric to trace' investigation workflow trustworthy.
The second reason is the investigation workflow itself, which is the whole point of unifying the signals. Metrics are cheap, always-on, and great for detection: an alert fires because p99 latency on the checkout service crossed its SLO. Traces are expensive, sampled, and great for diagnosis: they show which downstream call was slow. Historically these lived in separate tools with no bridge, so the on-call engineer saw the spike, then went hunting in the trace tool with a timestamp and a prayer. Exemplars — trace IDs stapled to metric buckets — close that gap: the spike on the dashboard links directly to a trace that is one of the slow requests in that bucket. Span metrics produce exemplars natively because they were computed from the very spans being linked. Detection and diagnosis become one continuous motion, which is the difference between a five-minute and a fifty-minute incident.
The third reason this rises to an architecture decision — rather than 'just turn on the connector' — is cardinality. Every unique combination of dimension values is a distinct metric time series, and time-series databases price you by active series. Choose dimensions carelessly — include a raw URL with IDs in it, a user ID, a full SQL string as the operation name — and the connector manufactures millions of series from a firehose of spans, and your metrics backend falls over. So the very feature that makes span metrics powerful (rich dimensions inherited from spans) is also the feature that will destroy your backend if ungoverned. Span metrics are therefore a governance problem as much as a plumbing one, and treating them casually is how teams learn about cardinality the expensive way.
The architecture: every piece explained
Top row: the pipeline. An instrumented app emits spans via OpenTelemetry as it always would — no extra metric code. Those spans arrive at the OTel Collector, and inside it the spanmetrics connector sits as a pipeline component that consumes the trace stream and produces a metric stream (a 'connector' bridges a traces pipeline into a metrics pipeline). For each span it extracts the configured dimensions — by default service name, operation (span) name, span kind, and status code, optionally augmented with selected span attributes — and updates the aggregates keyed by that dimension tuple. The spans continue, unmodified, to the trace backend in parallel; the connector is a tap, not a filter.
Middle row: the three RED signals and the link. Rate is a monotonic counter, conventionally calls_total (or traces.span.metrics.calls), incremented once per span per dimension tuple — divide by time in the query to get requests-per-second. Errors are captured by the status-code dimension: spans with an error status increment the same counter under a status_code=ERROR label, so error rate is a ratio of two label selections of the same series family. Duration is a histogram — the span's elapsed time is observed into latency buckets, from which the backend computes p50/p95/p99 per dimension. Exemplars attach a representative trace ID (and its timestamp) to histogram buckets and counters, so a point on the latency curve carries a pointer into the trace store — the metric-to-trace jump.
Bottom rows: destinations and governance. The metrics flow to a metrics backend (Prometheus, or any OTLP/PromQL-compatible TSDB) where dashboards and alerting rules live; the sampled spans flow to a trace backend (Tempo, Jaeger, a vendor) where exemplar links resolve. The ops strip names the disciplines that keep this alive: cardinality control (bounding the dimension set and normalizing high-cardinality values), dimension governance (which span attributes are allowed to become metric labels), histogram bucket design (buckets matched to your SLO thresholds so percentiles are accurate near the boundaries you alert on), and sampling awareness — because if the connector sees only sampled spans, the derived rates are undercounts, so the connector must sit before tail sampling in the pipeline to see every span.
A subtlety worth calling out is aggregation temporality — whether the emitted counters and histograms are cumulative (ever-growing totals) or delta (per-interval increments). The choice must match what the metrics backend expects: Prometheus-style scraping wants cumulative series and computes rates with rate(), while some OTLP push pipelines prefer delta temporality that the backend sums. Getting this wrong produces graphs that look plausible but are quietly halved, doubled, or sawtoothed across collector restarts. The connector and the exporter must agree on temporality, and the dashboards' rate math must match the temporality chosen — a small configuration detail that, when mismatched, manifests as the maddening class of bug where the numbers are wrong but never obviously broken, so nobody notices until a capacity decision is made on a figure that was off by a factor of two.
End-to-end flow
Trace the full loop on a checkout service with a 300ms p99 SLO. Every request produces a server span named by route template (POST /checkout, not the raw URL with a cart ID — that normalization is the cardinality discipline in action). Those spans stream into the collector. The spanmetrics connector, placed before the tail-sampling processor so it sees 100% of spans, increments calls_total{service=checkout, span_name="POST /checkout", status_code=OK} and observes each duration into the latency histogram, attaching an exemplar trace ID to the bucket the duration fell into.
Detection: a deploy at 14:00 introduces a slow database call. The Prometheus alerting rule computes p99 from the span-metrics histogram over a five-minute window and, at 14:03, sees it cross 300ms; it fires an SLO burn-rate alert. The on-call opens the latency dashboard — also driven by the span metrics — and sees p99 jump from 180ms to 620ms on the checkout endpoint precisely at the deploy marker, while rate (from calls_total) is flat and error rate (the status_code=ERROR ratio) is still near zero. So: not a traffic surge, not failures, purely latency regression.
Diagnosis via exemplar: the engineer clicks a point on the p99 curve in the spike. The exemplar attached to that histogram bucket carries a real trace ID from one of the slow requests; the click opens that exact trace in the trace backend. The trace shows the checkout span at 640ms, with 500ms of it inside a single database span whose statement is an un-indexed query the deploy introduced. Detection (metric alert) to root cause (specific trace, specific span) took two clicks and under a minute — because the metric and the trace were the same underlying spans, linked by exemplar. The fix (add the index / revert) is validated the same way: p99 on the span-metrics dashboard drops back under 300ms and the burn-rate alert clears.
Contrast the world without span metrics: the alert fires from a hand-written histogram whose buckets don't quite match the SLO, the dashboard shows a spike, and the engineer switches to a separate trace tool, searches by service and time range, wades through sampled traces hoping one is slow, and maybe finds the culprit in twenty minutes if the slow ones weren't sampled away. The architecture didn't make the database faster; it collapsed the distance between noticing and knowing.
One more property of the loop deserves emphasis: the numbers on the dashboard and the numbers in the trace are the same numbers. When the p99 metric said 620ms, it was a percentile over the exact population of span durations the connector observed, and the exemplar it handed back was a member of that population — not a correlated-by-timestamp guess. This is why the workflow is trustworthy in a way that separately-instrumented metrics never quite are: there is no possibility that the histogram counted requests the trace store scopes differently, because both were derived from one stream of spans. The engineer never has to wonder 'is the dashboard measuring the same thing as the trace?' — a question that, in a two-instrumentation world, quietly undermines confidence in every incident and is impossible to fully answer.