Observability / interactive experiment

Alert fatigue

Can one operator keep up? Send alerts, filter the noise, and see which incidents wait too long.

Explore · observe · explain
The on-call deskReady to explore
Alerts moving from monitoring through a filter into the operator queue Orange cards are noise; red incident cards need a response within twelve simulated seconds. Metrics below give the current state. 01 · MONITORING02 · FILTER03 · QUEUE04 · ON-CALL Alert source Score ≥ 20 0 waiting Ready for a page Filtered / grouped0 / 0 QUEUE HISTORY · 0–60 simulated seconds · scale 0–20 alerts (higher values capped)t = 0.0s
Noisy signalReal incidentHandledOutline turns red after the 12s deadline
Waiting pages0Excludes current service
Handled pages00 noisy pages handled
Incidents past deadline0Includes filtered real incidents
Operator utilization0%Share of elapsed time busy
0.0 / 60.0s
Replay0.0s recorded
Press Play. Follow the cards from the monitor to the filter, queue and operator.

Try a controlled experiment

  1. Run Noisy baseline. Watch the queue grow while the operator handles repeated warnings.
  2. Choose Grouped alerts and replay the same traffic. Compare noisy pages handled and missed deadlines.
  3. Set the severity threshold to 100. The queue empties—but real incidents are filtered too.
  4. Pause, send an incident, then step forward. Inspect its time in the queue.

Decision trace

    Model: fixed seeded arrivals, FIFO service, a 12-second response deadline, noise severity 10–75 and incident severity 60–100. 45% of noisy arrivals repeat the same warning key; grouping remembers the last paged signal. Severity ranges overlap, so aggressive filtering has a cost. These are explicit educational assumptions, not measured human response rates.

    Run a controlled alert experiment

    Begin with the displayed alert rate and false-positive share. Press Play and watch the arrivals, pages sent to the operator, and incidents waiting beyond the response deadline. Pause before changing a setting so the current queue is easy to compare. Increase Group repeated noise from zero and restart: repeated warning keys should produce fewer pages, while unique incidents still need attention. Next raise the Severity threshold. The page count may fall again, but inspect the missed-incident count because noisy and real severity ranges overlap.

    Change Handling time only after testing the filters. Longer service time makes the single operator a bottleneck even if the incoming alert rate is unchanged. Use the speed menu to inspect a difficult interval, then restart with the same seeded arrivals to compare two policies fairly. This model assumes one FIFO operator, fixed severity ranges, and a 12-second teaching deadline. It does not measure human fatigue, escalation behavior, or the quality of a real on-call team.

    Follow an alert through an actual queue

    Each arriving signal has an identity, a synthetic severity score, a birth time and an explicit state. It travels from the monitor to the gate, then is filtered, grouped or placed in the operator’s FIFO queue. One operator handles one page at a time. Orange cards are known synthetic noise; red cards are known incidents. A red outline marks an incident that has missed the fixture’s response deadline. The visible scene and counters come from those per-alert states, not a fixed animation that always reaches a successful conclusion.

    Change the five operating conditions

    Alert rate sets incoming signals per modeled minute. False-positive share controls the seeded probability of a noise signal. Severity threshold admits scores greater than or equal to the threshold after a 0.9-second travel interval. Group repeated noise suppresses a repeat with the same signal key when its last admitted page falls within the chosen number of seconds; zero disables grouping. Handling time sets service duration for newly started pages. Changes apply to future decisions in the current run; they do not retroactively undo a page or shorten work already started. Restart clears the run and uses the current settings.

    Compare a storm with repeated-noise grouping

    Noise storm sets ninety signals per minute, eighty-five percent noise, threshold twenty, no grouping and three-second handling. Run sixty modeled seconds and record queue length, handled pages and missed incidents. Choose the tuned grouping preset: it repeats those settings with thirty-second grouping. The seeded sequence is reproducible when the same configuration and actions are replayed, so compare the resulting backlog. Some noise has a unique key and cannot be grouped. A grouped repeat does not update the last admitted-page timestamp, so it does not indefinitely extend that suppression interval.

    See the cost of an aggressive severity gate

    Set Severity threshold to one hundred and restart. Generated incident scores range from sixty through ninety-nine, so no generated incident passes this threshold. Queue depth falls to zero, but missed-incident count rises. Inject incident creates a known severity-ninety-five incident at the current time, and the same gate applies to it. Lower the threshold to let it enter the queue. A missed incident means it was filtered or remained unfinished more than twelve seconds after birth; one completed after that deadline stays counted as missed. Low queue length alone is therefore not a sufficient success metric.

    Replay, branch and inspect the operator

    Play and Pause control progression; Step +1s advances twenty fifty-millisecond model steps. Restart clears the current history. Speed chooses half, normal, double or fourfold viewing pace without changing the service arithmetic. Replay seeks only through already recorded states. Resuming or stepping from an earlier state truncates the later branch and continues from that inspected point with the current controls. Inject incident also branches from the inspected state. The queue shows up to seven cards and an explicit remaining count; the queue-history line clips visually at twenty waiting pages, while the numeric queue counter remains exact. The event list retains only its twelve latest messages.

    Keep the operational assumptions visible

    The operator is a single fixed-capacity FIFO server with no prioritization, breaks, escalation or on-call handoff. Busy percentage measures occupied service time, not human stress or cognitive fatigue. The simulator knows which signals are noise and only groups that known-noise class; a real deduplication system cannot rely on perfect ground-truth labels. Thresholding can hide real incidents, and grouping is limited to the deliberately repeated warning key. Severity and false-positive probabilities are synthetic, not calibrated incident data. There is no production monitoring backend, alert delivery network, customer impact measurement or guaranteed paging policy. Use the experiment to discuss workload, lost signals and response deadlines together.

    Views are my own and do not represent any employer, past or present.