Symptoms, not causes
Alert on 'checkout error rate >2%', not 'Postgres connection count >800'. The cause-based alert pages whether or not it affects users; the symptom-based alert pages when users hurt. Causes go in dashboards, not pages.
Advertisement
Multi-window burn rate
Single threshold alerts are noisy. Use Google's burn-rate pattern: page if (5-min window burning > 14× AND 1-hr window burning > 14×) for fast-burn, separately for slow-burn. Catches both spikes and steady leaks; ignores transient blips.
Advertisement
Auto-resolve, not auto-page
If the condition self-recovers within 10 minutes, don't page. Aggregate into a 'this happened N times today' summary email. Most transient alerts shouldn't have woken anyone up.