Why architecture matters here
Recording rules matter architecturally because they move computation off the read path and onto a controlled write-time schedule, and that relocation is the whole point. In a monitoring system, reads are unpredictable and bursty — a human opens a dashboard, an incident triggers a flood of queries, dozens of alert rules evaluate on their own cadences — while writes are steady. If every read recomputes an expensive expression from raw samples, the cost of that expression is multiplied by an unbounded, spiky read rate. A recording rule fixes the cost at one evaluation per interval, no matter how many consumers there are, converting a variable many-times-per-interval cost into a constant once-per-interval one. That is a fundamental change in the system's load profile, not a minor optimization.
The defining trade-off is freshness for cost, and understanding it governs how you use rules. A recorded series is a snapshot taken at the last evaluation, so its data is at most one interval old; a query against raw data is as fresh as the latest sample. For almost all dashboards and alerts this staleness is irrelevant — nobody needs a five-minute rate computed to sub-second freshness — but it is real, and it means recording rules are the wrong tool for anything that must reflect the absolute latest sample. The interval is the knob: shorter intervals mean fresher recorded data at higher evaluation cost, longer intervals mean cheaper evaluation at the price of staleness. Choosing the interval is choosing where on that curve each rule sits.
The second architectural lever is cardinality, and recording rules cut both ways on it. Their best use is cardinality reduction: a rule that takes a metric with thousands of per-instance, per-path series and aggregates it down to a handful of per-service series produces a cheap, low-cardinality metric that dashboards and alerts can query trivially, while the expensive high-cardinality raw data still exists for ad-hoc drill-down. But a rule that preserves or multiplies labels can create new high-cardinality series and write them back every interval, permanently inflating the database. The architectural discipline is to use rules to summarize — to trade detail you do not need at query time for cheapness you do — and never to mint high-cardinality series casually.
The subtlest architectural property is that rules compose, and composition must be ordered. A recording rule can reference the output of another recording rule — you might record a per-instance rate, then record a per-service sum of those rates, then alert on the per-service value. Within an evaluation cycle those dependencies must run in order: the sum rule must evaluate after the rate rule it depends on, or it reads last interval's value and lags a step behind. This is why rules are organized into groups that evaluate their members sequentially, and why cross-group dependencies are fragile — groups run independently, so a rule in group B that depends on a rule in group A can always be one interval stale. Designing the group structure so dependent rules share a group, and ordering them correctly within it, is a real architectural decision that determines whether your derived metrics are consistent or subtly skewed.
The architecture: every piece explained
The top row is the precomputation pipeline. The raw series are the high-volume, often high-cardinality samples the monitoring system scrapes or ingests. The rule evaluator is the component that, on each group's schedule, runs the rule's query against those series. A recording rule is the definition itself: an expression plus a target metric name, saying 'evaluate this expression and store the result under this name'. The result is appended to the TSDB as a new series, indistinguishable from a scraped metric to everything downstream — it has a name, labels, and a value per timestamp.
The middle row is the consumers and the organizing structures. Dashboards read the cheap recorded series and render instantly instead of recomputing aggregations on every panel refresh. Alerting rules reference recorded metrics too, so an alert's condition evaluates against a precomputed value rather than a heavy raw expression — faster and less likely to time out. Rule groups are the unit of organization and scheduling: rules in a group evaluate in order, at the group's interval, so dependent rules can be sequenced correctly. Federation and remote write often consume recorded aggregates specifically — a central system pulls only the low-cardinality recorded summaries from many local servers rather than every raw series, which is what makes hierarchical monitoring scale.
The third row holds the timing machinery that makes or breaks correctness. The eval scheduler runs each group on its configured interval, typically with a per-group jitter so that not every rule fires on the same tick and hammers the database at once. Staleness and backfill is what happens when the evaluator falls behind: if a group's evaluation takes longer than its interval, iterations are missed, the recorded series develops gaps, and anything reading it sees stale or absent data during those gaps — a recording rule cannot retroactively fill the time it failed to evaluate.
The ops strip names the health signals. Rule evaluation duration tells you whether a group's work fits inside its interval; missed iterations tell you when it does not. Series churn tells you whether rules are creating or dropping series in ways that stress the database. And naming conventions are an operational concern as much as a stylistic one: recorded metrics form a namespace that dashboards and alerts hard-code, so a consistent, discoverable naming scheme is what keeps the derived-metric layer usable as it grows from ten rules to a thousand.
End-to-end flow
Walk a single evaluation. The scheduler reaches the tick for a rule group whose interval is thirty seconds. It begins evaluating the group's rules in order. The first rule's expression — say, a five-minute rate of an HTTP request counter, summed by service — is run against the raw series in the database. The evaluator computes one value per resulting series (one per service) at the current timestamp and appends those values to the TSDB under the rule's target name, for example a metric like a per-service request rate. The whole group finishes, the scheduler notes the duration, and the next tick is scheduled thirty seconds out. The expensive aggregation just happened once, in the background.
Now the read path, which is where the payoff lands. A dashboard panel that wants the per-service request rate no longer contains the heavy rate-and-sum expression; it contains a bare reference to the recorded metric. When a user opens the dashboard, the panel query is a simple series lookup — cheap, fast, and identical in cost whether one user or fifty open it simultaneously. An alerting rule that watches the same per-service rate likewise evaluates against the precomputed series, so its evaluation is trivial and reliable even during an incident when query load spikes. The raw counter still exists for someone who needs to drill into a single instance, but the common aggregate view is served from precomputed data.
Follow a dependency chain to see why ordering matters. Suppose one rule records a per-instance error rate, a second rule records the per-service sum of those per-instance rates, and an alert fires on the per-service value. If both rules live in the same group with the per-instance rule first, the evaluation cycle computes the per-instance rates, then immediately computes the per-service sum from the fresh values, and the alert sees a consistent number. If the two rules are split across groups, or ordered wrong within a group, the sum rule reads the previous cycle's per-instance values — it lags one interval, and during rapid change the per-service number is subtly inconsistent with its components. The fix is architectural, not a tuning tweak: keep dependent rules in one group and order them so each reads fresh inputs.
Finally, the failure moment: an evaluator that cannot keep up. A group accumulates rules over months until its combined evaluation takes forty seconds, but its interval is thirty. Now every cycle overruns; the scheduler either skips ticks or falls progressively behind, and the recorded series develop gaps where evaluations were missed. Dashboards reading those series show blank spots, and worse, alerts referencing them may go stale — an alert cannot fire on data that was never recorded. Crucially, recording rules do not backfill: when the evaluator recovers, it resumes from the current time; the missed interval stays empty forever. The operational response is to watch evaluation duration against interval as a first-class signal, split overloaded groups so each fits its interval, and lengthen intervals or simplify expressions where the work genuinely cannot be made to fit — because a recording rule that misses its schedule is not just slow, it silently stops producing the very data your alerts depend on.