Why architecture matters here
Architecture matters here because downsampling is a lossy transformation applied to the data your dashboards, alerts, and capacity models depend on, and the loss is not uniform — it destroys exactly the extremes that incidents live in. A 15-second spike that saturated a queue is a clear signal in raw data and can vanish entirely into a five-minute average. Whether your downsampled history tells the truth about the past depends on choosing the right aggregations up front, because once the raw data expires, the rollup is all you have; you cannot recompute a percentile you failed to store.
The economics are the whole point. Storage for metrics scales with cardinality times resolution times retention. Downsampling attacks the resolution-times-retention product: instead of paying to keep billions of high-resolution points for a year, you keep them at high resolution for days and at progressively lower resolution as they age, cutting long-term storage by one to two orders of magnitude. Query cost falls in lockstep — a dashboard showing a year of data reads a few hundred daily points instead of two million raw ones, so it loads in milliseconds instead of timing out. The architecture turns 'we can't afford to keep history' into 'history is nearly free'.
But the benefits are only real if the rollups are correct and transparent. Correct means the aggregation preserves the semantics you query — a counter rolled up by summing rates, a gauge by keeping min/max/avg, a histogram by summing buckets so percentiles remain computable. Transparent means dashboards and alerts do not need to know which tier they are reading; a query for the last year and a query for the last hour use the same expression, and the router picks the tier. Understanding the architecture is what lets you get both right, so that downsampling saves money without silently corrupting the story your metrics tell.
It is worth being precise about what kind of loss downsampling is, because it is often confused with two others it must be kept separate from. It is not sampling — dropping a random subset of points — which introduces statistical noise but preserves distribution shape; downsampling deterministically aggregates every point in a window, so it loses temporal resolution but not sample count. It is also not cardinality reduction — dropping label dimensions — which loses the ability to slice by that dimension; downsampling keeps every series but coarsens each one's time axis. Conflating these leads to bad decisions: teams reach for downsampling to fix a cardinality-driven cost problem, coarsen the time axis, and are baffled when storage barely moves because the cost was in the number of series, not the resolution. The architecture only pays off when you apply the right lossy transform to the right cost driver — downsampling for the resolution-times-retention product, dimension aggregation for cardinality — and often you apply both, in that order, at the coarse tiers.
The architecture: every piece explained
Top row: from firehose to rollups. Ingest accepts raw samples at full frequency and writes them to the raw tier — full resolution, short TTL (hours to days). The rollup engine runs continuously or on a schedule, reading raw samples and aggregating them over fixed windows into coarser tiers. The key design choice lives in the aggregations it computes and stores per window: not just the average, but min, max, sum, and count — the sufficient statistics from which most useful queries can be reconstructed and, critically, which compose so that an hourly rollup can be built from twelve five-minute rollups rather than from raw data.
Middle row: the tiers. Each tier trades resolution for retention. The 5-minute tier keeps medium resolution for weeks; the 1-hour tier keeps coarse resolution for months; the 1-day tier keeps very coarse resolution for years. A sample's journey is a cascade: raw rolls into 5m, 5m rolls into 1h, 1h rolls into 1d, and each source tier expires on its own TTL once the next tier has captured it. The query router is the piece that makes this usable: given a query's time range and step, it selects the finest tier that still covers the range and is affordable to scan — a one-hour view reads raw or 5m; a one-year view reads 1d — and it does so without the caller specifying a tier.
Bottom rows: policy and consumers. The retention policy declares, per tier, the resolution and the TTL — the two numbers that define the entire cost/fidelity curve; tuning them is how you dial the trade for your budget and your query patterns. Dashboards and alerts read through the router and should behave identically regardless of tier, which puts a hard requirement on the aggregations: a panel showing p99 latency must be able to compute p99 from whatever tier it lands on, which is only possible if the rollup stored histogram buckets rather than a pre-averaged number. The ops strip captures the discipline: choose aggregations that compose, validate that rollups match raw within tolerance, and make alerting tier-aware so a threshold means the same thing on 5m data as on raw.
End-to-end flow
Follow a latency metric from a service. At t=0, the service reports request latency as a histogram every 15 seconds; ingest writes each observation into the raw tier with full bucket detail. For the first 48 hours, dashboards querying 'p99 over the last hour' read the raw tier directly: the router sees a one-hour range at fine step, picks raw, sums the histogram buckets across the window, and computes p99 from the merged distribution — exact, high-resolution, and fast because an hour of raw data is small.
Every five minutes, the rollup engine reads the last five minutes of raw histograms and produces one 5m rollup: it sums the bucket counts (histograms compose by addition), records the window's min and max latency, the total count, and the sum of durations. It does not store a single pre-computed p99, because a percentile is not additive — you cannot average two windows' p99s to get the combined p99. By keeping the buckets, the 5m tier can still answer p99 correctly for any sub-range. After 48 hours the raw data for that period expires; the 5m tier now carries the load. A dashboard asking 'p99 over last week' routes to 5m, merges buckets across the week, and computes an accurate percentile — from data one-twentieth the size of raw.
The cascade continues. Hourly, twelve 5m rollups combine into one 1h rollup — buckets summed, min/max carried through (min of mins, max of maxes), counts and sums added — so the 1h tier is built from 5m, never re-reading raw. Daily, 24 hourly rollups combine into a 1d rollup the same way. A year later, an SRE investigating a capacity question asks for 'p99 latency, daily, over the last 12 months'. The router picks the 1d tier, reads ~365 points, merges the daily histograms across the requested granularity, and renders in milliseconds. Contrast the naive design that stored only per-window averages: that year-old data could show the mean drifting up but would be utterly unable to reconstruct p99 — the tail, where the incidents were, was aggregated away at the first rollup and is gone forever. The composing aggregations are what make the transparent, cheap, year-long query also a correct one.
The cascade also has an operational rhythm that the flow makes visible. Each tier's rollup job runs on a cadence tied to its window — the 5m rollup every five minutes, the hourly every hour — and each depends on the tier below being complete for the window it is aggregating. That creates a dependency chain with a small, bounded lag: a data point is queryable at full resolution instantly, appears in the 5m tier within a few minutes of its window closing, and propagates to daily within a day. A healthy pipeline keeps that lag well inside each tier's TTL, so raw data is always captured into 5m before it expires, and 5m into 1h, and so on. When a rollup job stalls — a slow query, a deploy, a backfill — the danger is not immediate wrongness but a race: if the raw tier expires the window before the delayed 5m rollup captures it, that resolution is lost permanently. This is why rollup lag is a first-class alert: it is the one metric whose breach silently and irreversibly destroys history, and it must be watched with the same seriousness as the ingest path itself.