Golden signals: watch a service saturate and recover

Last eight responses

Follow the workload before interpreting its charts

The queue and worker boxes describe an actual FIFO service simulation. Each request is assigned an ID and arrival time. It either starts on a free worker, waits in the bounded queue, or is immediately rejected when no service slot or waiting slot is available. A completed service generates a success or service-error response. A waiting request can time out before it starts service. At every displayed second, arrived requests equal responses plus pending work. The cards and charts are reductions of this request ledger; no sine wave or random latency is labeled as a percentile.

Set demand and service capacity

Requests per second controls demand from zero through fifty. Arrivals accumulate in fifty-millisecond steps, so rates above twenty may produce more than one arrival at a tick. Workers sets one through eight concurrent service slots. Service time sets the base duration, rounded up to the next fifty milliseconds. Queue slots bounds waiting requests, excluding active workers; zero permits immediate service but no waiting. Queue patience is measured from arrival and applies only while waiting. At its deadline the request times out before same-tick dispatch. An already serving request does not time out under this waiting-only rule.

Create an incident without changing the graph directly

Incident start and Incident duration choose when newly started services use Slowdown multiplier times the base service duration. Work already started retains its assigned completion time. Service failure percentage uses an integer remainder accumulator on completed services; rejected and queue-timeout responses are separate error classes. Remove incident sets its duration to zero. Restore defaults selects ten requests per second, two workers, one-hundred-millisecond service, forty queue slots, five-second patience, no service failures and a fivefold slowdown from second twenty through just before second forty. All input edits rebuild the deterministic run and pause it at zero.

Watch overload and delayed recovery

At default second ten, traffic is ten arrivals per second, the queue is empty and successful-request p99 is one hundred milliseconds. At second thirty, both workers are occupied, thirty-nine requests are waiting and trailing utilization is 99.5 percent. Of sixty responses in the trailing window, twenty-one are errors: ten rejections and eleven queue timeouts. The thirty-nine successful responses have a p99 of 5,400 milliseconds because their latency includes waiting. At second fifty the queue has drained, but p99 still includes slow earlier work in the trailing window. At second sixty it returns to one hundred milliseconds. An incident ending does not instantly erase backlog or measurement history.

Read all four signals with their denominators

Traffic counts arrivals in the last ten seconds divided by the elapsed window duration; the first seconds use a shorter window. Errors is failed responses divided by all responses in that same completion window, including rejection and queue-timeout responses. Latency is nearest-rank p99 over successful completed requests only, measured from arrival to response. It deliberately excludes failed responses; a service dropping slow work can therefore appear faster while its error signal gets worse. Saturation is the instantaneous waiting-request count. Utilization separately measures occupied worker time divided by available worker time across the trailing window. High utilization and a long queue are related but are not the same metric.

Inspect, replay and verify the accounting

Play and Pause control a sixty-second inspection. Next second advances one snapshot; Restart rewinds; Show final result completes the run. Speed selects half, normal or double pace and Inspect time scrubs. The diagram displays the first twelve waiting IDs and every worker; the exact waiting count remains visible even when more IDs do not fit. The response table shows the last eight outcomes with their arrival, service start, response time and latency. Exact evidence lists pending IDs, workers, counters and window error classes. At no observed responses, error percentage and p99 are unknown. At zero traffic workers are idle; unknown latency is not proof of a fast service.

Know the service model boundaries

The model is a single FIFO stage, not a complete application or monitoring stack. It has no retries, network delays, CPU scheduling, multiple service classes, downstream dependencies, autoscaling or real observability exporter. Fixed-duration workers process one request at a time. Completion frees workers before waiting deadlines and new admissions are handled at that tick; existing eligible waiters dispatch before fresh arrivals. All errors and durations are synthetic. The latency chart scales to the largest observed p99 so far; its labeled axis can expand, while traffic and error chart axes remain fixed. This dashboard can explain a capacity bottleneck, but cannot establish production alert thresholds or promise user outcomes.