Symptom-based, not cause-based
Bad: 'CPU > 80% on web-04'. Good: 'p95 latency > 500ms on /checkout for 5 minutes'. Symptoms map to user pain; causes generate noise. Alert on what users feel, not on what the box does internally.
Advertisement
Multi-window multi-burn-rate
Single threshold alerts (>1% error rate for 5min) fire too often. Google SRE pattern: alert if 5min burn-rate AND 1hr burn-rate both exceed thresholds proportional to your SLO. Fires only on real budget burns, not spikes.
Advertisement
Alert correlation
One root cause (DB outage) fires 50 downstream alerts. Tools: PagerDuty Event Intelligence, BigPanda, OpsRamp — cluster related alerts into one incident. Reduces page volume 5-10x without losing information.