Why architecture matters here
The architecture matters because overload without shedding does not degrade gracefully — it collapses. As arrival rate exceeds service rate, queues grow without bound; every message now waits behind a growing backlog, so latency climbs for all traffic, not just the excess. Worse, much of the queued work becomes worthless before it is served: a request whose client already timed out is processed and discarded, burning capacity on output nobody will read. This is congestive collapse — the server is 100% busy and its useful throughput is near zero. Shedding is the intervention that keeps useful throughput high by refusing to queue work it cannot serve in time.
Bidirectional streams make this sharper than plain request/response. A stateless HTTP server has a natural admission point — it can refuse a new connection — and each request is independent, so shedding one does not affect another. A bidi server holds long-lived, stateful streams that multiplex many logical messages; the connection is already open, so the shedding decision happens per-message inside the stream, not at connection admission. And because streams are long-lived, an overload can persist across many messages on the same connection, so the system must shed continuously and fairly rather than making a one-time admit/reject call.
Selectivity is the whole game, and it is why shedding is an architecture rather than a coin flip. Not all messages are equal: a trading system's order-cancel is critical while a market-data heartbeat is droppable; a video call's audio matters more than a presence update. Uniform random dropping under load discards critical and trivial work in proportion to their volume, which is exactly backwards, since the trivial work is usually the high-volume work. A priority-aware shedder drops the low tier first and protects the high tier, so the work that survives is the work that matters. That requires the system to know each message's priority — which is a design decision made long before the overload, in how messages are classified.
Fairness and cooperation are the final reasons. Under overload, one aggressive peer flooding the server can, without fairness, consume all the capacity and starve every well-behaved client — so shedding must allocate the survivors fairly across peers, not just globally by priority. And a drop that the client does not understand is a drop the client will immediately retry, adding load to an already-overloaded server in a retry storm that deepens the collapse. Signaling the shed — a 503 with a retry-after, a backoff hint on the stream, a closed low-priority substream — converts a blind drop into a cooperative backoff, so the client slows down instead of hammering. Detection, priority, fairness, and signaling together are what make shedding protect the server rather than merely relocate the pain, and each is a distinct architectural commitment.
The architecture: every piece explained
The overload detector is the trigger, and its choice of signal defines the whole system's behavior. Common signals are queue depth (are we backing up?), processing latency (are we slow?), and resource saturation (CPU, memory, connection count). The best signals are ones that rise before the server is fully saturated, giving the shedder time to act — queue depth and latency lead saturation, so they are preferred over a lagging CPU-at-100% signal. The detector outputs not just on/off but a degree of overload that scales how aggressively to shed.
The priority classifier ranks each message so the shedder knows what to drop first. Priority can come from the message type, the stream's class of service, the tenant's tier, or an explicit priority field. The key property is that the classification is cheap and available at shed time — you cannot afford to do expensive work to decide whether to drop something. Tiers are usually coarse (critical / standard / best-effort) because fine-grained priorities are hard to reason about under pressure.
Admission control is shedding at the edge — refusing new work before it enters the system — while the shed action handles work already inside. The action has several forms: silently drop a droppable message, return an explicit rejection (503 / RESOURCE_EXHAUSTED) so the peer knows, or close entire low-priority streams to reclaim their capacity. Which form fits depends on the message's semantics: perishable telemetry can be dropped silently, but a request the client is awaiting must be rejected explicitly so the client is not left hanging.
The peer signal turns a drop into cooperation. A retry-after hint, a backoff directive, or a flow-control message tells the client to slow down rather than retry immediately, preventing the retry storm that would otherwise deepen the overload. Fairness / quota ensures the survivors are allocated across peers — a per-peer rate limit or weighted fair queue so no single client monopolizes the reduced capacity. And the recovery ramp governs the way back: when the overload signal falls, the shedder does not instantly re-admit everything (which would immediately re-overload) but ramps admission up gradually, watching the signal to avoid oscillation. Together — detector, classifier, admission control, shed action, peer signal, fairness, and recovery ramp — these pieces form a control loop that keeps the server inside its capacity while doing the most valuable work it can, and each addresses a specific way that naive dropping fails.
End-to-end flow
Trace a server under a rising storm. Traffic across its bidi streams climbs and the overload detector, watching queue depth, sees the message queue begin to grow beyond its healthy band. It reports a mild overload degree. The shedder responds proportionally: it begins shedding only the best-effort tier — background telemetry, presence updates — dropping those messages silently because they are perishable and the peer does not await them. The critical and standard tiers pass untouched; latency for important work stays flat.
The storm intensifies. Queue depth and processing latency both climb further; the detector raises the overload degree. Now the shedder escalates: it continues dropping best-effort and begins shedding the standard tier's excess, but it does so with explicit rejections — returning RESOURCE_EXHAUSTED with a retry-after on those streams — because standard-tier senders are awaiting a response and must know their request will not be served. Fairness kicks in here: the rejection is applied per-peer against each peer's quota, so a single flooding client is shed heavily while well-behaved clients keep their fair share.
The peers receive the signal and cooperate. Clients seeing retry-after back off rather than retrying immediately, which pulls arrival rate down toward what the server can serve. The critical tier — the order-cancels, the control messages — has been protected throughout; the server never dropped a message that truly mattered, and it never tipped into collapse because it refused to queue work it could not serve in time. Useful throughput stayed high even as offered load exceeded capacity.
Now recovery. The backoff and shedding together bring queue depth back into the healthy band, and the detector's overload degree falls. The shedder does not fling the doors open — that would let the pent-up, still-elevated load slam back in and re-trip the detector, oscillating the server in and out of overload. Instead it ramps admission up gradually, first re-admitting standard-tier work, then best-effort, watching the queue signal at each step and holding if it starts to climb again. The system converges smoothly back to full service. Read as a whole, the flow is a feedback control loop: the detector measures pressure, the shedder actuates by dropping the least valuable work with peer cooperation, and the recovery ramp closes the loop without oscillation — which is exactly why selective shedding degrades gracefully where uniform dropping or no dropping at all would not.