Why architecture matters here
Cardinality matters because it's the single biggest driver of observability cost and performance, and cardinality explosions are among the most common and damaging observability failures -- silently ballooning cost and degrading (or crashing) the metrics system. The series count (cardinality) drives everything: storage (each series stored/indexed), cost (metrics systems price on cardinality -- so high cardinality is expensive), and query performance (more series to scan -- slower, more memory, potentially OOM-ing the metrics backend). A cardinality explosion (from a high-cardinality label) can balloon the cost (10x, 100x), degrade queries (slow dashboards), or crash the metrics system (OOM from too many series) -- and it's often silent (a label added innocently -- a user_id, a request_id -- quietly exploding the series count) until the cost or the outage appears. So managing cardinality (avoiding high-cardinality labels, detecting explosions, setting guardrails) is central to a cost-effective, performant observability system, and cardinality explosions are a common, damaging failure to prevent. For anyone operating metrics (essentially every production system), understanding cardinality is essential.
The every-combination-is-a-series insight is the crucial mechanical understanding, and it's why cardinality explodes multiplicatively. In a dimensional metrics system, a time series is uniquely identified by the metric name plus the full set of label key-value pairs. So each distinct combination of label values is a separate series. This means the cardinality is the product of the label value counts: a metric with labels method (say 5 values), status (say 10 values), and endpoint (say 20 values) has up to 5 x 10 x 20 = 1,000 series. This is manageable. But add a high-cardinality label -- user_id (a million values) -- and it multiplies: 1,000 x 1,000,000 = a billion series -- an explosion. The multiplicative nature is the trap: each label multiplies the series count by its value count, so a single high-cardinality label (with many values) explodes the total (multiplying by that large count). This is why high-cardinality labels (user_id, request_id, IP, timestamp, arbitrary strings -- each with many/unbounded values) are so dangerous: they multiply the series count by their large value count, exploding the cardinality. And it's why bounded labels (few values -- method, status, region) are safe: they multiply by a small count. Understanding that every label combination is a separate series (so cardinality is the product of label value counts, and a high-cardinality label multiplies it explosively) is understanding the core mechanic and why label choice is so consequential.
And the put-high-cardinality-detail-elsewhere principle is the crucial design discipline, resolving the tension between wanting detail and avoiding cardinality. You often want high-cardinality detail (per-user, per-request, per-IP information -- for debugging, analysis). The mistake is putting it in metrics (as labels -- exploding the cardinality). The discipline is recognizing that the three observability signals (metrics, logs, traces) have different cardinality characteristics and purposes. Metrics are for aggregatable, low-cardinality measurements (counts, rates, latencies -- aggregated across bounded dimensions) -- NOT for high-cardinality detail (which explodes them). Logs are for high-cardinality event records (each event a record -- with arbitrary detail, high cardinality is fine -- logs are built for it). Traces are for high-cardinality request flows (each trace a request, with detailed spans -- high cardinality is fine). So high-cardinality detail (user_id, request_id, per-request specifics) belongs in logs or traces (built for it), NOT metrics (which need low cardinality). And exemplars bridge them: a metric (low-cardinality -- e.g., a latency histogram) can carry exemplars (links to representative traces -- e.g., a link to a trace of a slow request in the high-latency bucket) -- so you get the aggregate view (the metric) plus the ability to drill into specific examples (the linked traces) -- without putting the high-cardinality detail in the metric. This -- keeping metrics low-cardinality (aggregatable measurements), putting high-cardinality detail in logs/traces (built for it), and linking them with exemplars -- is the crucial design discipline (getting the detail you need without exploding metric cardinality), and understanding it is understanding how to have both low-cardinality metrics and high-cardinality detail.
The architecture: every piece explained
Top row: the concept and impact. What is cardinality: the number of unique time series -- each a distinct combination of the metric's label values (cardinality = the product of the label value counts). Cardinality explosion: a high-cardinality label (many values -- user_id, request_id) multiplying the series count explosively (a single such label exploding the total). Storage and cost: the series count drives storage (each series stored/indexed) and cost (metrics systems price on cardinality) -- so high cardinality is expensive. Query performance: more series to scan and aggregate -- slower queries, higher memory (potentially OOM-ing the backend) -- so high cardinality degrades performance.
Middle row: labels and control. Bad labels: high-cardinality labels to avoid -- user_id, request_id, IP address, timestamp, arbitrary strings (each with many/unbounded values -- exploding cardinality). Good labels: bounded, low-cardinality labels -- method, status code, region, service (few, bounded values -- safe). Detection: finding the offending labels (identifying which labels/metrics are driving the cardinality -- via cardinality analysis tools, per-label series counts) -- to fix explosions. Limits and drop rules: guardrails -- cardinality limits (capping the series per metric), drop/relabel rules (dropping or aggregating away high-cardinality labels) -- preventing explosions.
Bottom rows: the signal split and linking. Metrics vs logs vs traces: put the detail in the right signal -- metrics for low-cardinality aggregatable measurements, logs and traces for high-cardinality detail (built for it) -- so high-cardinality data goes to logs/traces, not metrics. Exemplars: linking metrics to traces -- a low-cardinality metric (e.g., a latency histogram) carrying exemplars (links to representative traces) -- so you get the aggregate (metric) plus drill-into-examples (traces) without high-cardinality metric labels. The ops strip: label design (designing metric labels for low cardinality -- bounded labels, no high-cardinality data -- the primary discipline), monitoring cardinality (monitoring the series counts and cardinality -- catching explosions early, before the cost/outage), and budgets (cardinality budgets -- limits per metric/team -- governing the cardinality, attributing the cost).
End-to-end flow
Trace a cardinality explosion and its fix. A team adds a label to a request-count metric to track requests per user -- http_requests_total{..., user_id="..."}. This seems helpful (per-user request counts) but user_id is high-cardinality (a million users) -- so the metric explodes from its previous ~1,000 series (method x status x endpoint) to ~a billion series (multiplied by the million user_ids). The consequences appear: the metrics system's memory balloons (a billion series to store/index -- potentially OOM-ing it), the cost spikes (priced on cardinality -- a billion series is enormously expensive), and queries slow to a crawl (scanning a billion series). The team detects it (cardinality analysis showing the user_id label as the culprit -- the metric's series count exploded), and fixes it: they remove the user_id label from the metric (restoring it to low cardinality) -- and, since they still want per-user insight, they put that detail in logs/traces (logging each request with the user_id -- high cardinality is fine in logs) instead of the metric. The metric returns to low cardinality (fast, cheap), and the per-user detail lives in logs (where high cardinality belongs). The explosion (from the high-cardinality user_id label) was fixed by removing it from the metric and routing the detail to logs.
The detection and exemplar vignettes show the discipline. A detection case: the team monitors cardinality proactively (tracking the series counts per metric -- watching for growth) -- so when a new label starts exploding a metric's cardinality (a deployment adding a high-cardinality label), they catch it early (the cardinality monitoring alerting on the growth) -- before it balloons the cost or crashes the system -- and fix it (removing/bounding the label) promptly. The proactive cardinality monitoring caught the explosion early. An exemplar case: the team wants to drill from a latency metric (a histogram -- low cardinality) into specific slow requests. Rather than adding high-cardinality labels (request_id -- exploding the metric), they use exemplars: the latency histogram carries exemplars (links to representative traces -- e.g., a trace of a slow request in the high-latency bucket) -- so from the metric (the aggregate latency view), they can click through to example traces (the specific slow requests) -- getting the drill-down without high-cardinality metric labels. The exemplars linked the low-cardinality metric to high-cardinality traces (the best of both).
The label-design and budget vignettes complete it. A label-design case: the team establishes label-design discipline -- metrics use only bounded, low-cardinality labels (method, status, region -- few values), and high-cardinality data (user_id, request_id, IPs) is never a metric label (it goes to logs/traces) -- so metrics stay low-cardinality by design (avoiding explosions). A budget case: the team sets cardinality budgets (limits per metric/team -- and drop rules enforcing them) -- so a team's metrics can't explode the cardinality beyond their budget (the guardrails preventing runaway cardinality), and the cardinality cost is attributed (per team -- accountability). The consolidated discipline the team documents: keep metrics low-cardinality (bounded labels only -- method, status, region; never high-cardinality data -- user_id, request_id, IPs, timestamps), put high-cardinality detail in logs/traces (built for it) not metrics, use exemplars to link metrics to traces (drill-down without high-cardinality labels), monitor cardinality proactively (catch explosions early -- before the cost/outage), set guardrails (cardinality limits, drop rules) and budgets (per metric/team -- governance, attribution), and detect offending labels when explosions occur -- because cardinality (every label combination a separate series -- the product of label value counts) is the single biggest driver of observability cost and performance, and high-cardinality labels explode it multiplicatively, so keeping metrics low-cardinality (and routing high-cardinality detail to logs/traces) is essential to a cost-effective, performant observability system.