Why architecture matters here
SLM batching matters because batching is essential to efficient model serving throughput (even for small models), and continuous batching is the key technique -- while SLMs' cheapness and edge deployment shift the considerations. Serving models efficiently requires batching (running one request at a time underutilizes the hardware -- so batching fills it, amortizing overhead, maximizing throughput). This applies to SLMs too (a server-hosted SLM serving many requests benefits from batching -- filling the hardware). And continuous batching (requests joining/leaving per step -- keeping the batch full) is the key technique for generation (dramatically better throughput than static batching). But SLMs have specifics: they're cheaper (less compute per token -- so the overhead and batching efficiency matter proportionally more), and they often run at the edge/on-device (where batching is less relevant -- a single user -- and single-request latency dominates). So for server-hosted SLM serving, batching (especially continuous batching) is essential to throughput; for edge/on-device SLMs, the considerations differ (batch of one, latency-focused). Understanding SLM batching (the core batching techniques, and the SLM-specific nuances) is understanding how to serve SLMs efficiently.
The continuous-batching insight is the key technique, and it's why it dramatically beats static batching for generation. Generation (autoregressive -- producing tokens one at a time, for a variable number of tokens) creates a problem for static batching: requests in a batch generate different numbers of tokens (some finish quickly, some produce long outputs) -- so with static batching (run the batch until all finish), the batch is held until the longest request finishes (the already-finished requests occupying their slots but doing nothing -- wasting capacity -- and new requests waiting for the whole batch to finish before starting). This wastes a lot of capacity (the finished requests' idle slots, the waiting requests). Continuous batching solves this: it operates per decode step (each step generating one token for each active request) -- and between steps, requests can join and leave: a finished request leaves (freeing its slot immediately -- not waiting for the batch), and a waiting request joins immediately (taking the freed slot -- starting its generation) -- so the batch is continuously full (every slot doing useful work every step -- no idle finished requests, no waiting) -- dramatically improving throughput (the hardware continuously utilized) and reducing latency for waiting requests (they join as soon as a slot frees -- not waiting for the whole batch). This continuous batching (per-step join/leave -- keeping the batch full) is the key technique for efficient generation serving (much better than static batching's wait-for-the-longest) -- and it's used in modern serving (vLLM, TGI, etc.). Understanding continuous batching (per-step join/leave, keeping the batch full -- versus static batching's waste) is understanding the key batching technique for generation.
And the SLM-specific reality (cheapness and edge) is what distinguishes SLM batching from LLM batching, shifting the considerations. SLMs differ from LLMs in ways that affect batching. Cheapness: an SLM has less compute per token (it's small) -- so per token, the overhead (the fixed costs -- scheduling, kernel launches, the batching machinery) is proportionally larger (relative to the small compute), and the memory-bandwidth-bound decode phase's efficiency matters. So batching efficiency (amortizing the overhead, using the hardware well) matters proportionally more for SLMs (the overhead being a larger fraction of the cheap compute) -- efficient batching is important. Edge/on-device: SLMs often run at the edge or on-device (a phone, a laptop -- for latency, privacy, cost) -- where there's typically a single user (a batch of one -- so batching across requests is less relevant, and single-request latency dominates -- the concern being fast single-request generation, not throughput across many requests). So for edge/on-device SLMs, batching is less relevant (batch of one) and the focus shifts to single-request latency (and techniques like speculative decoding for single-request speedup). This SLM-specific reality (cheapness -- batching efficiency matters more for server-hosted; edge/on-device -- batch of one, latency-focused) distinguishes SLM batching from LLM batching (where the model is expensive and server-hosted throughput dominates) -- so the batching considerations differ by SLM deployment (server-hosted: continuous batching for throughput; edge: batch of one, latency). Understanding the SLM-specific reality (cheapness, edge deployment -- shifting the considerations) is understanding how SLM batching differs from LLM batching.
The architecture: every piece explained
Top row: the problem and approaches. The problem: per-request serving is inefficient (underutilizing the hardware) -- so batch requests together (filling the hardware, amortizing overhead). Static batching: wait for a batch to fill, run it (until all finish), then the next -- with the problem that the batch is held until the longest request finishes (wasting capacity on finished requests). Continuous batching: requests join and leave the batch per decode step (a finished request leaves, a waiting request joins -- keeping the batch full) -- dramatically better throughput. Throughput vs latency: the core tradeoff -- larger batches (more throughput -- but potentially more per-request latency); a timeout balances filling the batch against latency.
Middle row: phases and limits. Prefill vs decode: the two generation phases -- prefill (processing the prompt -- compute-heavy, parallel across the prompt tokens) and decode (generating tokens one at a time -- memory-bandwidth-bound) -- with different batch characteristics (handled differently in the batching). KV cache + memory: each request's KV cache (the model's per-request state -- growing with the sequence) consumes memory -- so the batch size is limited by the available memory (the KV caches of the batched requests fitting in memory) -- a key constraint. Small-model specifics: SLMs have less compute per token -- so the overhead and batching efficiency matter proportionally more (the overhead a larger fraction of the cheap compute). Edge + on-device: SLMs often run at the edge/on-device (a single user -- a batch of one -- so batching is less relevant; single-request latency dominates).
Bottom rows: config and comparison. Dynamic batching config: the batching configuration -- the max batch size (limited by memory/latency) and the timeout (how long to wait to fill a batch before running -- balancing filling against latency) -- the tuning knobs. vs LLM batching: SLM batching is similar to LLM batching (continuous batching, the same throughput/latency tradeoff, KV-cache limits) -- but the model is cheaper (shifting the compute/overhead balance -- overhead matters more) and often edge-deployed (batch of one) -- similar techniques, different balance. The ops strip: batch tuning (tuning the batch configuration -- max batch size, timeout -- for the throughput/latency balance and the memory limit), SLA (meeting the latency SLA -- the batching's latency impact balanced against throughput -- so requests meet their latency requirements), and monitoring (monitoring the serving -- throughput, latency, batch sizes, queue depths, memory -- for performance and tuning).
End-to-end flow
Trace continuous batching serving SLM requests. A server hosts an SLM, receiving many concurrent requests. With continuous batching: at each decode step, the server generates one token for each active request in the batch. When a request finishes (generates its final token), it leaves the batch immediately (freeing its slot) -- and a waiting request joins immediately (taking the freed slot -- starting its prefill/decode). So the batch stays continuously full (every slot doing useful work every step -- no idle finished requests, no waiting for the batch) -- maximizing throughput (the hardware continuously utilized). Compare to static batching: the batch would be held until the longest request finished (finished requests idling, new requests waiting) -- wasting capacity. The continuous batching (per-step join/leave) kept the batch full -- dramatically better throughput than static batching -- serving the SLM requests efficiently. And the batch size is limited by the KV-cache memory (the batched requests' KV caches fitting in memory) -- so the server batches up to the memory limit.
The tradeoff and memory vignettes show the constraints. A throughput-latency case: the team tunes the batching for their SLA -- a larger max batch size and a longer timeout give more throughput (more requests batched) but potentially more latency (requests waiting to fill the batch, and the larger batch's per-step time); a smaller batch/shorter timeout gives lower latency but less throughput. They balance it for their SLA (enough throughput while meeting the latency requirement) -- the throughput-latency tradeoff tuned. A memory case: the batch size is limited by the KV-cache memory (each batched request's KV cache consuming memory -- so the batch can only be as large as the memory allows). The team manages this (the KV-cache memory limiting the batch -- and techniques like paged KV cache improving the memory efficiency, allowing larger batches) -- the memory constraint on batching.
The edge and comparison vignettes complete it. An edge case: for an on-device SLM (a single user on a phone), batching across requests is irrelevant (a batch of one -- a single user's requests) -- so the focus shifts to single-request latency (fast generation for the one user -- e.g., speculative decoding for single-request speedup) -- the edge deployment changing the considerations (batch of one, latency-focused -- not throughput batching). A comparison case: the team notes that server-hosted SLM batching is like LLM batching (continuous batching, the throughput/latency tradeoff, KV-cache limits) -- but the SLM is cheaper (so the overhead and batching efficiency matter proportionally more) -- similar techniques, different balance. The consolidated discipline the team documents: batch requests for efficient server-hosted SLM serving (per-request is inefficient -- batching fills the hardware), use continuous batching (requests joining/leaving per decode step -- keeping the batch full -- dramatically better throughput than static batching), navigate the throughput-latency tradeoff (batch size, timeout -- tuned for the SLA), respect the KV-cache memory limit (the batch size limited by memory -- paged KV cache helping), account for SLM specifics (cheapness -- batching efficiency matters more; edge/on-device -- batch of one, latency-focused), tune the batching config and monitor the serving -- because batching is essential to efficient server-hosted model serving throughput (even for SLMs), with continuous batching the key technique, while SLMs' cheapness and edge deployment shift the considerations (batching efficiency matters more; edge is batch-of-one, latency-focused).