Requirements worth pinning down first
Four numbers determine the architecture, and getting them approximately right at the start matters more than any later optimisation.
Active series count. Not samples per second -- series. A series is one unique combination of metric name and label values, and it is the unit everything indexes, shards and remembers. Ten million active series is a large installation; a hundred million is a specialist one.
Sample rate. Scrape or push interval times series count. Ten million series at fifteen-second resolution is about 670,000 samples per second, sustained, forever.
Retention by resolution. Raw samples for days or weeks, downsampled aggregates for months or years. These are different retentions on different data, and conflating them makes the storage estimate wrong by an order of magnitude.
Query shape. Dashboards, which are predictable and repetitive, versus ad-hoc investigation, which is unpredictable and expensive, versus alerting rules, which run constantly and must never be starved by the other two.
One asymmetry belongs in the requirements explicitly: ingest availability matters more than query availability. A query that fails can be retried; a sample that is dropped is gone, and it is gone precisely during the incident when someone will want it. Design so that the query tier can fail without affecting ingest.
The data model — series identity and why cardinality is the scaling variable
A sample is a triple: series identity, timestamp, value. The series identity is a metric name plus a set of key-value labels -- http_requests_total{service="checkout", method="POST", status="200", pod="checkout-7f9"} -- and it is hashed into a stable series ID that the whole system uses as its primary key.
The number of series is the product of the distinct values of every label. Four services times five methods times eight status codes times fifty pods is 8,000 series for one metric name. Add one label whose values are unbounded -- a user ID, a request ID, a full URL path with identifiers in it, an email address -- and the product is unbounded too. This is cardinality explosion, and it is the dominant failure mode of every metrics platform ever built, because the cost is not in the samples but in the index and the per-series memory that must be held for each active series.
The system design consequence is that the platform cannot treat cardinality as the user's problem. It has to enforce limits, because a single mislabelled metric in one service can take down storage for everyone. Concretely: reject or drop series beyond a per-tenant active-series limit, cap labels per series and label-value length, and expose the current count against the limit so teams can see themselves approaching it. The management of cardinality as a practice is a topic of its own; the enforcement of it is an architectural requirement.
Collection — push and pull
Pull means the platform scrapes an endpoint each target exposes. Service discovery gives it the target list, so the set of things being monitored is explicit and a target that disappears is immediately visible; scrape success is itself a health signal; and a runaway service cannot flood the platform faster than the scrape interval allows. It struggles with anything short-lived -- a batch job may not exist when the scraper arrives -- and with targets behind network boundaries the scraper cannot cross.
Push means agents or applications send samples to a receiver. It handles short-lived jobs, serverless functions and network-isolated environments naturally, and it decouples emitters from platform topology. In exchange the platform loses the implicit inventory -- a service that stops pushing is indistinguishable from one that was decommissioned -- and gains the need for authentication and per-source rate limiting at the edge, because now anyone can send anything.
Most real platforms end up with both: pull as the primary mechanism for long-lived services, a push gateway for batch work, and an agent per host that collects locally and forwards. The convergent answer in current practice is a standard telemetry protocol spoken by a vendor-neutral collector, which lets the collection layer be swapped independently of the storage layer -- worth insisting on, because storage backends get replaced more often than instrumentation does.
The ingest tier
Receivers should be stateless and horizontally scalable, behind a load balancer, doing five things: authenticate the sender, validate and normalise the payload, enforce per-tenant limits, batch, and hand off to the next stage. Statelessness is the point -- anything requiring per-series state at the edge prevents free scaling and makes deployments risky.
The significant design decision is whether a durable buffer sits between ingest and storage. A log such as Kafka, partitioned so that all samples for a series land in one partition, buys several things at once: storage can be restarted, upgraded or briefly overwhelmed without dropping data; a second consumer can be added to backfill a new cluster or feed a streaming aggregation without touching the write path; and ingest spikes are absorbed by disk rather than by memory. It costs an extra hop of latency, an extra system to operate, and a second copy of the data in flight.
For platforms below a few hundred thousand samples per second, direct write to storage with local retry buffers on the agents is simpler and adequate -- the agents themselves become the buffer. Above that, or wherever storage must be replaced without a maintenance window, the queue pays for itself. The deciding question is usually not throughput at all: it is whether you can tolerate losing data during a storage deployment.
Whatever the choice, ordering must be preserved per series and need not be preserved globally, which is exactly what partitioning by series ID gives. Out-of-order samples within a series are a genuine complication for compressed time-series storage, and while modern engines accept a bounded amount of lateness, the ingest design should not create it gratuitously.
Sharding
Shard by series ID, hashed. Not by metric name -- a single metric name like a request counter can be a large fraction of all traffic, and name-based sharding puts it on one node. Not by tenant alone -- tenants differ in size by orders of magnitude. Hashing the full series identity spreads load evenly because the label combinations are numerous and roughly uniform in weight.
The property to preserve is stability: a given series must always map to the same shard, so that its history is contiguous and a query for it touches one place. Consistent hashing with virtual nodes provides that while limiting how much data moves when the shard count changes -- a plain modulo remaps almost everything on a resize, which for a system holding weeks of history is not a resize but a migration.
Within a shard, partition by time: fixed-length blocks, commonly a couple of hours for recent data, compacted into larger blocks as they age. This gives a clean retention story -- expiry is deleting whole blocks rather than tombstoning individual samples -- and it bounds query work, since a range query only opens the blocks it overlaps.
The two-dimensional layout, series across shards and time within them, is what makes both common query shapes efficient: 'this series over a long range' reads one shard sequentially, and 'many series over a short range' fans out across shards and touches one or two blocks in each.
The storage engine
Three components, and the design is broadly the same across the well-known implementations.
A head block in memory holds the most recent window of samples, appended to per series, backed by a write-ahead log so a crash loses nothing. This is where all writes go and where most queries are answered from, since dashboards mostly ask about now.
An inverted index maps each label key-value pair to a sorted list of series IDs holding it. A query with matchers such as {service="checkout", status="500"} is answered by intersecting two posting lists -- the same machinery as a text search engine, for the same reason: it turns 'find the matching series' from a scan into a set operation. The index is the expensive part of high cardinality; the samples themselves compress beautifully, and the index does not.
Immutable blocks on disk hold older data: compressed chunks per series plus that block's own index, written once, never modified, deleted whole at retention. Compaction merges small blocks into larger ones, which reduces the number of index lookups a long-range query performs.
Compression is what makes the economics work, and the technique is worth knowing by name. Delta-of-delta encoding on timestamps exploits the fact that samples arrive at a near-fixed interval, so after two levels of differencing most values are zero and cost a bit or two. XOR encoding on float values exploits the fact that consecutive readings of the same gauge are numerically close, so their bit patterns share leading and trailing bits, leaving a handful of meaningful ones. Together these bring storage to roughly one to two bytes per sample against sixteen for a naive timestamp-and-double encoding, which is the difference between a metrics platform and a bankruptcy. Sizing follows directly: 670,000 samples per second at ~1.5 bytes is about 87 GB per day before replication.
The query path
A query is four stages: parse and plan, resolve series via the index, read and decode chunks, aggregate.
Resolution is the intersection of posting lists for each label matcher, and it is where a regular-expression matcher on a high-cardinality label becomes expensive, because it cannot be answered by a direct lookup and must scan the label's values. Reading is bounded by how many series survived resolution times how many chunks each has in range. Aggregation then collapses those series according to the query.
In a sharded deployment the plan must push down: each shard resolves and aggregates locally, returning partial results that a coordinator merges. This is the same reasoning as any distributed query engine -- move the aggregate, not the rows -- and it works for the additive aggregates that dominate metrics. Sum, count, min and max merge trivially; averages must be carried as sum and count and divided at the end; quantiles cannot be merged exactly at all, which is why metrics systems compute them from bucketed histograms whose buckets are additive counters, and why a quantile of an average of quantiles is meaningless.
Limits belong in the query path as a first-class feature, not as an afterthought: maximum series touched, maximum samples scanned, wall-clock timeout, and concurrent queries per tenant. Without them a single unbounded query -- someone graphs a high-cardinality metric with no matchers over thirty days -- consumes the memory of every shard it touches. Alerting evaluation should run in a separate query pool from interactive dashboards for the same reason, so that a heavy dashboard cannot delay the rules that page someone.
Rollups and multi-resolution storage
A thirty-day chart at fifteen-second resolution asks for 172,800 points per series to render a few hundred pixels. Precomputing coarser resolutions is not an optimisation, it is a requirement above a certain retention.
Run continuous aggregation that writes one-minute, five-minute and one-hour rollups alongside the raw data. The essential detail is which aggregates to store: keep sum, count, min and max, never a bare average. Sum and count reconstruct the average at any coarser granularity and can themselves be rolled up further; a stored average cannot be combined with another average without weights, so a chain of averages produces silently wrong numbers. This is the same additivity constraint that governs pushdown in the query path, and violating it is one of the most common defects in home-grown metrics systems.
Then make resolution selection automatic. The query planner picks the coarsest series whose resolution still gives adequate points for the requested range, so users write one query and get raw data for a one-hour window and hourly rollups for a one-year window without knowing the difference. Exposing resolution as a user-facing choice guarantees someone will pick raw over a year and time out.
Retention is then per resolution: raw for two weeks, one-minute for three months, one-hour for two years. The rollups are small enough that long retention on them costs little, which is what makes multi-year trend analysis affordable.