A metrics system is a database with an unusually lopsided workload. Writes arrive continuously, in time order, at a rate that does not vary much and never stops -- millions of samples per second is an ordinary size for a mid-scale platform. Reads are rare by comparison but expensive when they happen, because a dashboard panel asks for an aggregate over thousands of series across hours of history, and thirty panels refresh at once. Almost nothing is updated, nothing is deleted individually, and the value of a sample decays sharply with age. That shape is specific enough that general-purpose databases lose badly to purpose-built ones, and it is what every design decision below follows from. The concepts underneath -- what a counter is, how cardinality behaves, when to downsample -- are covered in this site's observability articles; this is the system design.
Requirements worth pinning down first
Four numbers determine the architecture, and getting them approximately right at the start matters more than any later optimisation.
Active series count. Not samples per second -- series. A series is one unique combination of metric name and label values, and it is the unit everything indexes, shards and remembers. Ten million active series is a large installation; a hundred million is a specialist one.
Sample rate. Scrape or push interval times series count. Ten million series at fifteen-second resolution is about 670,000 samples per second, sustained, forever.
Retention by resolution. Raw samples for days or weeks, downsampled aggregates for months or years. These are different retentions on different data, and conflating them makes the storage estimate wrong by an order of magnitude.
Query shape. Dashboards, which are predictable and repetitive, versus ad-hoc investigation, which is unpredictable and expensive, versus alerting rules, which run constantly and must never be starved by the other two.
One asymmetry belongs in the requirements explicitly: ingest availability matters more than query availability. A query that fails can be retried; a sample that is dropped is gone, and it is gone precisely during the incident when someone will want it. Design so that the query tier can fail without affecting ingest.
The data model — series identity and why cardinality is the scaling variable
A sample is a triple: series identity, timestamp, value. The series identity is a metric name plus a set of key-value labels -- http_requests_total{service="checkout", method="POST", status="200", pod="checkout-7f9"} -- and it is hashed into a stable series ID that the whole system uses as its primary key.
The number of series is the product of the distinct values of every label. Four services times five methods times eight status codes times fifty pods is 8,000 series for one metric name. Add one label whose values are unbounded -- a user ID, a request ID, a full URL path with identifiers in it, an email address -- and the product is unbounded too. This is cardinality explosion, and it is the dominant failure mode of every metrics platform ever built, because the cost is not in the samples but in the index and the per-series memory that must be held for each active series.
The system design consequence is that the platform cannot treat cardinality as the user's problem. It has to enforce limits, because a single mislabelled metric in one service can take down storage for everyone. Concretely: reject or drop series beyond a per-tenant active-series limit, cap labels per series and label-value length, and expose the current count against the limit so teams can see themselves approaching it. The management of cardinality as a practice is a topic of its own; the enforcement of it is an architectural requirement.
Collection — push and pull
Pull means the platform scrapes an endpoint each target exposes. Service discovery gives it the target list, so the set of things being monitored is explicit and a target that disappears is immediately visible; scrape success is itself a health signal; and a runaway service cannot flood the platform faster than the scrape interval allows. It struggles with anything short-lived -- a batch job may not exist when the scraper arrives -- and with targets behind network boundaries the scraper cannot cross.
Push means agents or applications send samples to a receiver. It handles short-lived jobs, serverless functions and network-isolated environments naturally, and it decouples emitters from platform topology. In exchange the platform loses the implicit inventory -- a service that stops pushing is indistinguishable from one that was decommissioned -- and gains the need for authentication and per-source rate limiting at the edge, because now anyone can send anything.
Most real platforms end up with both: pull as the primary mechanism for long-lived services, a push gateway for batch work, and an agent per host that collects locally and forwards. The convergent answer in current practice is a standard telemetry protocol spoken by a vendor-neutral collector, which lets the collection layer be swapped independently of the storage layer -- worth insisting on, because storage backends get replaced more often than instrumentation does.
The ingest tier
Receivers should be stateless and horizontally scalable, behind a load balancer, doing five things: authenticate the sender, validate and normalise the payload, enforce per-tenant limits, batch, and hand off to the next stage. Statelessness is the point -- anything requiring per-series state at the edge prevents free scaling and makes deployments risky.
The significant design decision is whether a durable buffer sits between ingest and storage. A log such as Kafka, partitioned so that all samples for a series land in one partition, buys several things at once: storage can be restarted, upgraded or briefly overwhelmed without dropping data; a second consumer can be added to backfill a new cluster or feed a streaming aggregation without touching the write path; and ingest spikes are absorbed by disk rather than by memory. It costs an extra hop of latency, an extra system to operate, and a second copy of the data in flight.
For platforms below a few hundred thousand samples per second, direct write to storage with local retry buffers on the agents is simpler and adequate -- the agents themselves become the buffer. Above that, or wherever storage must be replaced without a maintenance window, the queue pays for itself. The deciding question is usually not throughput at all: it is whether you can tolerate losing data during a storage deployment.
Whatever the choice, ordering must be preserved per series and need not be preserved globally, which is exactly what partitioning by series ID gives. Out-of-order samples within a series are a genuine complication for compressed time-series storage, and while modern engines accept a bounded amount of lateness, the ingest design should not create it gratuitously.
Sharding
Shard by series ID, hashed. Not by metric name -- a single metric name like a request counter can be a large fraction of all traffic, and name-based sharding puts it on one node. Not by tenant alone -- tenants differ in size by orders of magnitude. Hashing the full series identity spreads load evenly because the label combinations are numerous and roughly uniform in weight.
The property to preserve is stability: a given series must always map to the same shard, so that its history is contiguous and a query for it touches one place. Consistent hashing with virtual nodes provides that while limiting how much data moves when the shard count changes -- a plain modulo remaps almost everything on a resize, which for a system holding weeks of history is not a resize but a migration.
Within a shard, partition by time: fixed-length blocks, commonly a couple of hours for recent data, compacted into larger blocks as they age. This gives a clean retention story -- expiry is deleting whole blocks rather than tombstoning individual samples -- and it bounds query work, since a range query only opens the blocks it overlaps.
The two-dimensional layout, series across shards and time within them, is what makes both common query shapes efficient: 'this series over a long range' reads one shard sequentially, and 'many series over a short range' fans out across shards and touches one or two blocks in each.
The storage engine
Three components, and the design is broadly the same across the well-known implementations.
A head block in memory holds the most recent window of samples, appended to per series, backed by a write-ahead log so a crash loses nothing. This is where all writes go and where most queries are answered from, since dashboards mostly ask about now.
An inverted index maps each label key-value pair to a sorted list of series IDs holding it. A query with matchers such as {service="checkout", status="500"} is answered by intersecting two posting lists -- the same machinery as a text search engine, for the same reason: it turns 'find the matching series' from a scan into a set operation. The index is the expensive part of high cardinality; the samples themselves compress beautifully, and the index does not.
Immutable blocks on disk hold older data: compressed chunks per series plus that block's own index, written once, never modified, deleted whole at retention. Compaction merges small blocks into larger ones, which reduces the number of index lookups a long-range query performs.
Compression is what makes the economics work, and the technique is worth knowing by name. Delta-of-delta encoding on timestamps exploits the fact that samples arrive at a near-fixed interval, so after two levels of differencing most values are zero and cost a bit or two. XOR encoding on float values exploits the fact that consecutive readings of the same gauge are numerically close, so their bit patterns share leading and trailing bits, leaving a handful of meaningful ones. Together these bring storage to roughly one to two bytes per sample against sixteen for a naive timestamp-and-double encoding, which is the difference between a metrics platform and a bankruptcy. Sizing follows directly: 670,000 samples per second at ~1.5 bytes is about 87 GB per day before replication.
The query path
A query is four stages: parse and plan, resolve series via the index, read and decode chunks, aggregate.
Resolution is the intersection of posting lists for each label matcher, and it is where a regular-expression matcher on a high-cardinality label becomes expensive, because it cannot be answered by a direct lookup and must scan the label's values. Reading is bounded by how many series survived resolution times how many chunks each has in range. Aggregation then collapses those series according to the query.
In a sharded deployment the plan must push down: each shard resolves and aggregates locally, returning partial results that a coordinator merges. This is the same reasoning as any distributed query engine -- move the aggregate, not the rows -- and it works for the additive aggregates that dominate metrics. Sum, count, min and max merge trivially; averages must be carried as sum and count and divided at the end; quantiles cannot be merged exactly at all, which is why metrics systems compute them from bucketed histograms whose buckets are additive counters, and why a quantile of an average of quantiles is meaningless.
Limits belong in the query path as a first-class feature, not as an afterthought: maximum series touched, maximum samples scanned, wall-clock timeout, and concurrent queries per tenant. Without them a single unbounded query -- someone graphs a high-cardinality metric with no matchers over thirty days -- consumes the memory of every shard it touches. Alerting evaluation should run in a separate query pool from interactive dashboards for the same reason, so that a heavy dashboard cannot delay the rules that page someone.
Rollups and multi-resolution storage
A thirty-day chart at fifteen-second resolution asks for 172,800 points per series to render a few hundred pixels. Precomputing coarser resolutions is not an optimisation, it is a requirement above a certain retention.
Run continuous aggregation that writes one-minute, five-minute and one-hour rollups alongside the raw data. The essential detail is which aggregates to store: keep sum, count, min and max, never a bare average. Sum and count reconstruct the average at any coarser granularity and can themselves be rolled up further; a stored average cannot be combined with another average without weights, so a chain of averages produces silently wrong numbers. This is the same additivity constraint that governs pushdown in the query path, and violating it is one of the most common defects in home-grown metrics systems.
Then make resolution selection automatic. The query planner picks the coarsest series whose resolution still gives adequate points for the requested range, so users write one query and get raw data for a one-hour window and hourly rollups for a one-year window without knowing the difference. Exposing resolution as a user-facing choice guarantees someone will pick raw over a year and time out.
Retention is then per resolution: raw for two weeks, one-minute for three months, one-hour for two years. The rollups are small enough that long retention on them costs little, which is what makes multi-year trend analysis affordable.
High availability and replication
The idiom that works for metrics is replicated ingest with query-time deduplication. Two or more independent ingest paths receive or scrape the same targets and write to separate storage replicas. Nothing coordinates them, so there is no consensus protocol in the write path and no shared failure. At query time the reader merges replicas and deduplicates, preferring whichever has data for each interval.
This works because metrics tolerate the inconsistency it produces. Two scrapers hit the same endpoint a few hundred milliseconds apart, so their samples have slightly different timestamps and possibly slightly different values -- and for time-series aggregation that is immaterial. Deduplication picks one replica per series and falls back when it has gaps, which is exactly the behaviour wanted when one replica was down for ten minutes.
The alternative -- a quorum-replicated distributed database with strong consistency -- is available and rarely worth it here. It puts a coordination protocol in front of the highest-throughput path in the system to buy a guarantee the workload does not need. Reserve strong consistency for the metadata: the tenant registry, limits, and rule definitions, which are small and where correctness genuinely matters.
For long-term storage, the common pattern is to ship compacted blocks to object storage and let a query layer read them directly, keeping only recent data on local disk. That decouples retention from cluster sizing entirely and makes historical capacity a storage bill rather than a capacity plan.
Multi-tenancy and guardrails
A shared platform needs isolation in three places, and the failures are different in each.
At ingest: per-tenant limits on active series, samples per second, labels per series and label-value length, enforced by rejecting rather than by buffering. Rejection with a clear error and a visible metric is far kinder than silent acceptance followed by a platform-wide outage a week later, and it puts the feedback where the fix is.
At query: per-tenant concurrency, series and sample scan limits, and timeouts, so that one team's exploratory query cannot exhaust shared memory.
At storage: separate retention per tenant, and ideally separate physical shards for the largest tenants so that their compaction and their incidents stay theirs.
The single most valuable guardrail is a cardinality report per tenant: which metric names and which label keys account for the most active series, updated continuously and visible to the teams that own them. Most cardinality explosions are accidents -- a label added in a hurry that happens to contain an identifier -- and they are trivially fixed once someone can see which label is responsible. Without the report, the platform team finds out by paging at three in the morning and then has to work out whose metric it was.
Failure modes, and build versus buy
Cardinality explosion is the one that takes the platform down: index and per-series memory grow until storage nodes exhaust memory, and recovery is slow because the offending series are already persisted. Guardrails at ingest are the only real defence, plus the ability to drop a label or a metric by configuration without a deploy.
Query stampedes: many dashboards refreshing on the same boundary. Jitter refresh intervals, cache dashboard query results briefly, and keep alerting in a separate pool.
Ingest backpressure: when storage slows, agents buffer and then drop, and the gap lands exactly during the incident. A durable queue converts this from data loss into lag.
Clock skew: samples timestamped by the emitter arrive out of order or in the future. Clamp future timestamps, bound acceptable lateness, and prefer server-side timestamps where the semantics allow.
On build versus buy: the honest position is that this is a well-served category and the reasons to build are narrow. Mature open-source implementations already embody every design decision above, and hosted offerings remove the operational burden entirely. Build when your scale is genuinely beyond what those handle, when a regulatory constraint forbids the alternatives, or when metrics are your product. Otherwise the valuable work is in the layer above -- instrumentation standards, cardinality governance, useful dashboards and alerts that mean something -- which is where the actual reliability benefit comes from and which no vendor can supply.