Why architecture matters here
SLO architecture matters because SLOs done well align engineering incentives with user experience. Metrics that reflect user pain drive fixes; metrics that don't waste effort. Get SLO definition wrong and you either alert on nothing meaningful or wake on-call constantly for problems users never notice.
Cost matters. Higher SLO targets cost more infrastructure. The right SLO trades reliability cost against feature velocity.
Reliability comes from budget-based policy. When budget is consumed, ship less; when budget is healthy, ship more. This clean policy turns reliability decisions into data-driven ones.
The architecture: every piece explained
Walk the diagram top to bottom.
User Journey. The operations users care about: "load a page," "search," "submit an order." SLIs measure these.
SLI. Service Level Indicator — a numeric quality signal. "Fraction of requests that succeed within 500ms."
SLO. Target percentage. "99.9% of requests over 30 days must be good."
Error Budget. 1 - SLO. If SLO is 99.9%, budget is 0.1% = 43 min per 30 days.
Burn Rate Alerts. Multi-window (fast + slow), multi-threshold. Page when 5% of budget burns in 1 hour; ticket when 10% burns in 6 hours.
Metrics + Traces. Source data for SLI computation. Prometheus histograms, sampled traces.
SLO Store. Sloth, Nobl9, Grafana OSS SLO, or in-house. Compute SLI, current SLO status, budget remaining.
Ownership. Named team owns each SLO. Not "SRE owns them all."
Policy on Budget. When budget healthy, ship features. When burning, focus on reliability. Codified rules.
Postmortem + Review. Every breach reviewed; policies updated; SLIs refined.
End-to-end SLO lifecycle
Trace a scenario. Product team owns "search" journey. SLI: fraction of search requests with p99 < 500ms and success=true. SLO: 99.5% over 30 days.
Error budget: 0.5% × 30 days = 3.6 hours.
Weeks 1-2: healthy. Budget consumed 20% (43 min). All fine.
Week 3: incident. Bad deploy caused 3% of search errors for 30 min. Budget consumed: 54 min. Cumulative: 40%.
Burn rate alert fires: "10% budget in 1 hour" threshold. Page on-call. Team rolls back deploy. Postmortem: what SLI signaled? What tests missed?
Week 4: quiet. Small burns. Cumulative: 45%.
End of window. 60% budget consumed. Budget policy: 25-75% burn = normal ship pace. Team ships next feature.
Alternative scenario: 90% budget consumed. Policy: freeze features except reliability work. Ship rate slows; reliability investment increases. Budget restores over next window.