Why it matters

Iteration scheduling enables continuous batching + priority. Foundation of throughput.

Advertisement

The architecture

Decide per step: admit, run, evict.

Priority per request.

Preempt if needed.

Iteration schedulingPer stepadmit, run, evictPriorityper requestPreemptif neededOrca introduced; enables continuous batching and preemption
Iteration scheduling.
Advertisement

How it works end to end

Orca-style scheduling.

Per-step decisions.

Preempt low-priority.