Why architecture matters here
The naive architecture — evaluate every alert rule against every series on every scrape — is exactly what Prometheus does, and it works brilliantly up to the point where humans can no longer write and maintain the rules. A mid-size platform has tens of thousands of service-endpoint-region combinations, each with latency, traffic, error, and saturation signals, each with its own daily and weekly rhythm. Hand-tuning a threshold per series is impossible; a single global threshold per metric class is guaranteed to be simultaneously too tight for some series and too loose for others. The result is the on-call experience every engineer knows: pages that fire on Monday morning traffic ramps, silence during the slow-burn regression.
Anomaly detection changes the cost model. Instead of paying with human rule-writing time, you pay with compute and architectural complexity: state per series, a training pipeline, a feedback mechanism. That trade is only worth making when the series count is large enough that rules have failed — which is why the architecture matters so much. A design that keeps per-series state small (a few hundred bytes of seasonal coefficients rather than raw history), keeps detection incremental (O(1) per datapoint, not a window re-scan), and keeps the expensive statistics offline in a nightly trainer can watch millions of series on a handful of nodes. A design that gets any of those wrong melts.
There is also an organizational reason the architecture matters: trust. Operators abandon anomaly systems after a few bad weeks of noise, and once abandoned they never come back. The pipeline pieces that look optional — persistence gating, severity scoring, correlation, shadow mode, the feedback store — are precisely the pieces that manage the false-positive rate, and they are the difference between a system teams rely on and a dashboard nobody opens. Treat the alert-quality machinery as load-bearing, not as polish.
The architecture: every piece explained
The series router is the front door. Metrics arrive as a firehose — remote-write from Prometheus, OTLP from collectors — and the router hashes each unique series (metric name plus label set) to a detector shard. Consistent hashing keeps a series pinned to the same shard so its state stays local; shard count scales horizontally with series cardinality. Each shard owns the feature window for its series: a small ring buffer of recent values from which it computes rolling statistics — rate of change, short-window mean and variance — incrementally, without rescanning.
The baseline store holds the learned model of normal, one compact record per series. The workhorse is seasonal decomposition: a daily and weekly seasonal profile (typically 288 five-minute buckets for the day, 7 day-of-week multipliers), a slowly-adapting trend term, and a robust spread estimate (median absolute deviation rather than standard deviation, so past spikes don't inflate tolerance). Holt-Winters variants update these online; a nightly model trainer refits them from the metrics warehouse with the luxuries streaming can't afford — outlier-resistant regression, holiday calendars, and deploy markers so a step change from a release becomes the new normal instead of a week-long anomaly.
Detection itself is an ensemble of cheap detectors, because no single test covers all failure shapes. A residual z-score against the seasonal baseline catches level shifts. An EWMA control chart catches slow drifts the z-score smooths away. A rate-of-change detector catches cliffs. Some teams add an isolation forest or matrix-profile pass for the exotic shapes, but the boring detectors find the vast majority of real incidents. Each detector emits a score; the anomaly scorer combines them with a persistence requirement — the condition must hold for k of the last n evaluations — and a severity estimate based on how far outside tolerance the series sits and how much traffic it represents. Downstream, the correlation engine groups anomalies that share time windows and topology (same service, same dependency edge, same host rack) into a single incident, and the alert gateway dedupes, routes by team config, and pages.
End-to-end flow
Follow one datapoint through the pipeline. A checkout service's http_requests_total error-rate series produces a value at 14:35: 0.8% errors, against a typical Tuesday-afternoon baseline of 0.05%. The remote-write stream delivers it to the router, which hashes the series ID and forwards it to shard 41 — the shard that has owned this series since it first appeared. The shard appends the value to the series' ring buffer, updates its rolling statistics in constant time, and looks up the baseline record: expected value 0.05%, MAD-derived tolerance 0.07%, trend flat, no deploy marker active.
The detectors fire in sequence, each a few microseconds. The residual z-score is enormous — the value sits more than ten tolerance-widths above baseline. The EWMA chart confirms the level shift began three evaluations ago. The persistence gate checks its window: this is now the third consecutive evaluation outside tolerance, which crosses the 3-of-5 requirement, so the shard emits an anomaly event — series ID, severity high (large deviation, high-traffic service), first-seen timestamp 14:25, detector votes attached.
The correlation engine receives the event and finds company: in the same two-minute window it has anomalies from the payment gateway's latency series, the checkout service's downstream-call errors, and four pod-level series that all share one Kubernetes node pool. Topology metadata links checkout to payments; the engine merges them into one incident anchored on the earliest anomaly, tags the probable origin (payments latency moved first), and hands one enriched alert to the gateway. The gateway checks team routing, sees no active silence, verifies this incident isn't a duplicate of an open one, and pages the payments on-call — one page, not seven. When the responder later marks the page as a true positive and notes the cause, that label lands in the feedback store, and the nightly trainer will exclude the outage window from the next baseline refit so the incident doesn't teach the model that 0.8% errors are normal on Tuesdays.
Notice what the flow never does: it never scans history at request time, never consults a central coordinator on the hot path, and never blocks one series' evaluation on another's. Every per-datapoint decision uses only shard-local state — the ring buffer, the baseline record, the persistence window — which is what keeps the pipeline's ingest-to-decision latency in seconds and its horizontal scaling boring. The cross-series work, correlation, happens once per anomaly rather than once per datapoint, downstream of the gate that has already discarded the overwhelming majority of evaluations as normal.