Why it matters

Admission control shapes SLO guarantees. Understanding shapes serving.

Advertisement

The architecture

Estimate load.

Reject if over threshold.

429 or queue.

Admission controlLoad estimatequeue depth + tokensDecisionadmit / reject / queueResponse202 / 429 / acceptLittle's Law + queue depth signal overload; better to shed than degrade all
Admission control.
Advertisement

How it works end to end

Signals: queue depth, KV pool, batch size.

Reject with 429 + Retry-After.

Reserve for high-priority.