Why it matters

Alert fatigue is the top cause of on-call burnout and missed incidents. Getting alerting right protects the team and the service.

Advertisement

The architecture

Symptom alerts: fire when users are affected (error rate up, latency high).

Cause alerts: fire on symptoms of impending issue (disk 90%, CPU high). Only alert if action needed now.

Alerting designSymptom alertsuser-visible impactCause alertsaction-required onlyPlaybook + escalationfor each alertEvery page must be actionable — otherwise, dashboard it or ticket it
Alert types and follow-through.
Advertisement

How it works end to end

Alert quality: every alert needs a documented playbook and clear action.

SLO-based alerting: alert when error budget consumption spikes. Fewer, more meaningful alerts.

Multi-level: page on-call for urgent; ticket for less urgent; log-only for informational.