Why architecture matters here
The reason this matters is a statistical fact that trips up nearly everyone: percentiles do not average. If web server A has a p99 of 200ms and server B has a p99 of 800ms, the fleet's p99 is not 500ms — it could be anywhere depending on how many requests each served and how their distributions overlap. Any dashboard that computes a per-instance p99 and then averages, maxes, or means those numbers across instances is displaying a quantity with no statistical meaning. The same trap swallows time-rollups: you cannot average one-minute p99s into an hourly p99. The only correct way to get an aggregate quantile is to combine the underlying distributions and re-estimate — which requires that the distribution be stored in a mergeable form. That form is the histogram.
The intuition for why is worth internalizing: a percentile is a rank statistic, and ranks are not linear in the underlying counts. Two servers each serving very different traffic volumes contribute to the fleet distribution in proportion to their request counts, not equally, so an unweighted average of their percentiles ignores the very weighting that defines the aggregate. Even a request-weighted average of percentiles is wrong, because the position of the 99th-percentile request in the combined distribution depends on the shape of both tails, information a single number per server has already thrown away. Only the full per-bucket counts retain enough of the distribution to recombine correctly.
So the architecture matters because it is the difference between latency numbers that are true and latency numbers that merely look plausible. When SLOs, alerts, and capacity decisions ride on p99, a subtly-wrong p99 is worse than none — it builds confidence in a lie. Histograms give you the mergeable substrate: because each bucket is a monotonic counter, summing bucket k across every instance yields the fleet's count of requests at-or-below that bound, and interpolating across the summed buckets yields a real fleet quantile. The same additivity makes time-rollups honest — sum the per-scrape bucket deltas over the window, then estimate once.
The cost side is equally architectural. A histogram with 20 buckets, exported per endpoint, per status code, per region, is not one metric — it is 20 times the product of those label cardinalities, each a stored time series. Latency histograms are, in most systems, the single largest driver of metric cardinality and therefore of TSDB cost. That tension — you want fine buckets and many dimensions for insight, and every one of them multiplies storage — is why bucket layout and label discipline are not tuning afterthoughts but the central design decision.
The architecture: every piece explained
Top row: recording. Instrumentation wraps the operation and calls observe(duration). The library consults the bucket layout — a sorted list of upper bounds (le, 'less than or equal') — and increments every bucket whose bound is ≥ the observed value; classic histograms are cumulative, so the counters are monotone and each bucket counts everything at-or-below it. It also adds the duration to a running _sum and bumps _count. These per-series counters — one series per le value plus sum and count — are the entire on-the-wire representation. A collector scrapes or receives them; because they are counters, the backend works with rates and deltas over intervals.
Middle row: storage and query. In the TSDB, each bucket is an independent time series identified by its le label. This is what makes histograms mergeable: to aggregate across instances or endpoints you sum by (le) — adding the per-bucket counts — and to roll up over time you take the counter increase across the window. The quantile estimate is computed at query time: to find p99, compute the target rank (0.99 × total count), walk the cumulative buckets to find the one where the running count first crosses that rank, and linearly interpolate within that bucket's [lower, upper] bound. The estimate is only as precise as that bucket is narrow — the structural error of fixed histograms. SLOs and alerts ride directly on buckets, often more robustly than on estimated quantiles: a '99% of requests under 300ms' SLO is answered exactly by dividing the count in the ≤300ms bucket by the total, no interpolation needed, and multi-window burn-rate alerts are just ratios of bucket counts.
Bottom rows: the modern alternatives. Native/exponential histograms replace hand-picked bounds with buckets whose boundaries grow geometrically at a configured relative resolution; a single schema covers microseconds to minutes, buckets are created only where data lands, and merges stay exact because the boundary scheme is shared. Sketches — t-digest and DDSketch — take a streaming approach: DDSketch guarantees a bounded relative error on every quantile (ideal for latency, where you care about percentage accuracy across orders of magnitude) and merges losslessly, while t-digest packs extreme accuracy at the tails into a small footprint. The ops strip is the recurring work: choosing bucket bounds that bracket your SLOs, budgeting the cardinality those buckets cost, understanding interpolation error, and visualizing the whole distribution as a heatmap rather than trusting one p99 line.
End-to-end flow
Trace a p99 from request to dashboard. A checkout service handles a request in 143ms. Its histogram middleware calls observe(0.143); the library increments every cumulative bucket with le ≥ 0.143 — the ≤0.25, ≤0.5, ≤1, ≤2.5, … buckets all tick up, the ≤0.1 and ≤0.05 buckets do not — and adds 0.143 to the sum, 1 to the count. Across a minute this service on this instance records 12,000 requests, populating its bucket counters. The collector scrapes the counters every 15 seconds; the backend stores 14 bucket series plus sum and count for this instance.
An SRE opens the fleet latency panel spanning 40 instances. The query is histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket[5m]))). Read it inside-out: rate(...[5m]) turns each cumulative bucket counter into a per-second increase over the window; sum by (le) adds those rates across all 40 instances, producing the fleet's cumulative distribution — this is the mergeability payoff, one honest fleet distribution rather than 40 un-combinable p99s. histogram_quantile(0.99, ...) then finds the bucket where the cumulative fraction crosses 0.99 and interpolates: say the ≤0.5 bucket holds 98.3% and the ≤1.0 bucket holds 99.4%, so p99 lands between 500ms and 1000ms, interpolated to ~820ms. That ~820ms carries the structural caveat: the true p99 could be anywhere in [500ms, 1000ms]; linear interpolation assumes uniform density in the bucket, which tail latencies rarely honor.
Now the accuracy failure. Suppose the real p99 matters for a 900ms SLO and the coarse ≤0.5→≤1.0 gap makes the estimate untrustworthy right at the decision boundary. Two fixes exist. Add bounds — insert ≤0.6, ≤0.7, ≤0.8, ≤0.9, ≤1.0 — so the crossing bucket is narrow and interpolation is tight; the cost is five more series per label combination. Or switch the metric to a native exponential histogram at, say, 5% relative resolution, so the bucket covering ~820ms is inherently narrow without hand-tuning and the whole range stays covered. Meanwhile the SLO alert sidesteps interpolation entirely: 'fraction of requests over 900ms' is computed by summing the count above the ≤0.9 boundary and dividing by total — an exact ratio if 0.9 is a real bucket bound, which is the strongest reason to place bucket bounds on your SLO thresholds. Finally the SRE switches the panel to a heatmap: instead of one p99 line, time on the x-axis and buckets on the y-axis with color for count reveals a bimodal distribution — a fast path and a slow path — that any single percentile would have flattened into a misleading average.