Why architecture matters here
Metrics fail on cardinality, not on missing data. Every unique combination of labels creates a time series; a user_id label on a busy service creates millions. Storage grows exponentially; queries slow to a crawl; retention shrinks.
The architecture matters because it controls this. Recording rules pre-aggregate; cardinality budgets enforce label economy; sharding across TSDBs prevents any single instance from becoming the bottleneck.
With the pieces mapped, you can grow metrics coverage without paying an exponential storage bill.
The architecture: every piece explained
The top strip is the pipeline. Service instruments code with a client library. Exporter exposes metrics in Prometheus format. Scrape / push collects them; scrape is the norm, push is for short-lived jobs. TSDB stores the time series — Prometheus for single-node, Thanos/Mimir/Cortex for horizontal scale.
The middle row is the metric types + economy. Counters + gauges are the primitive types. Histograms + summaries let you compute percentiles from buckets. Cardinality budget per metric prevents label explosion. Recording rules precompute common aggregates so queries at read time are cheap.
The lower rows are consumption. Alert rules use PromQL to define conditions; Alertmanager routes to on-call. Dashboards visualize; SLO dashboards close the loop. Ops handles retention, downsampling for long-term storage, and on-call tuning.
End-to-end flow
End-to-end: a service exposes /metrics. Prometheus scrapes every 15 seconds. Metrics include http_requests_total with labels (route, status_code) — bounded cardinality. A histogram (le buckets) captures latency. Recording rules precompute p95 per minute. Dashboards show request rate, error rate, and latency (RED). Alert fires when error rate > 2% for 5 minutes; Alertmanager routes to the service on-call. Long-term storage keeps hourly aggregates for a year via Thanos downsampling.