SGLang, in depth: RadixAttention, cache-aware scheduling, structured decoding and running it in production

The runtime's architecture

The SGLang runtime splits serving into processes. The HTTP server and TokenizerManager accept requests and turn text into token ids. A Scheduler process per data-parallel rank owns the waiting queue, the radix tree and the decision of what runs in each forward pass. Tensor-parallel workers execute the model. A DetokenizerManager turns output ids back into streamed text. Keeping tokenization and detokenization out of the scheduler's process keeps Python overhead off the critical path between GPU steps.

The scheduler also overlaps its own CPU work with GPU execution: while batch N runs on the GPU, it prepares batch N+1. This overlap scheduler is on by default, and --disable-overlap-schedule turns it off, which is useful when you are debugging and want strictly sequential steps.

RadixAttention: the KV cache as a prefix tree

Every token the model has processed has keys and values in every layer. Those KV entries depend only on the tokens before them. So if two requests start with the same 3,000 tokens, the KV for those tokens is identical, and the second request can reuse it instead of recomputing it. RadixAttention keeps finished requests' KV in GPU memory, indexed by a radix tree, which is a trie whose edges are labelled with runs of tokens rather than single tokens. Each node maps its token run to the KV slots that hold it.

On arrival, the scheduler walks the tree with the request's tokens and finds the longest matching prefix. Only the rest is prefilled. When a request finishes, its tokens are inserted into the tree. If they diverge partway along an existing edge, that edge is split in two. Nodes in use by running requests carry a reference count and cannot be evicted. When memory runs out, the scheduler evicts unreferenced leaves, by default the least recently used (--radix-eviction-policy accepts lru or lfu). A simplified version:

class Node:
    def __init__(self):
        self.children = {}      # first token of edge -> Node
        self.key = []           # token run on the edge into this node
        self.slots = []         # KV slot index per token in key
        self.ref = 0            # running requests using this node
        self.last_used = 0.0

def match_prefix(root, tokens):
    node, i, slots, path = root, 0, [], []
    while i < len(tokens) and tokens[i] in node.children:
        child = node.children[tokens[i]]
        n = common_prefix_len(child.key, tokens[i:])
        if n < len(child.key):          # partial edge match: split so the match ends on a node
            child = split(child, n)
        slots += child.slots
        path.append(child)
        i += n
        node = child
    return slots, path                  # caller increments ref on every node in path

def evict(root, needed):
    leaves = heap_of_unreferenced_leaves(root, key=lambda n: n.last_used)
    freed = 0
    while freed < needed and leaves:
        leaf = heappop(leaves)
        free_kv_slots(leaf.slots)
        freed += len(leaf.slots)
        parent = detach(leaf)
        if parent.ref == 0 and not parent.children:
            heappush(leaves, parent)    # the parent may become evictable

Two details distinguish this from block-based prefix caching. --page-size defaults to 1, so a prefix can match to the exact token instead of being rounded down to a full block. The cache also holds any finished sequence, not just prefixes that were declared up front, so reuse happens without any change to the client. Prefix caching explains the block-hash approach for comparison. Larger pages can make kernels more efficient, but they coarsen matching. If you raise the page size, measure the hit rate again.

Scheduling for cache hits

A cache is only useful if requests that share a prefix run while that prefix is still resident. The --schedule-policy flag controls ordering of the waiting queue. In current source the default is fcfs (first come, first served). lpm, longest prefix match, sorts waiting requests by how much of them is already cached. This raises hit rates and batches similar requests together, at the cost of fairness. dfs-weight schedules in depth-first order over the tree, so that a whole subtree is served together. Other values include priority, lof and random. Defaults have changed between releases, so check --help on the version you deploy rather than relying on a blog post, including this one.

Long prompts are prefilled in chunks (--chunked-prefill-size tokens at a time) so that one 100,000-token prompt does not freeze decoding for everyone else. --max-running-requests caps the batch, and --schedule-conservativeness controls how much KV room the scheduler holds back for running requests to grow into before it admits new ones. Set it too low and requests are preempted and recomputed. Set it too high and batches stay small.

Worked example: an agent with a long system prompt

Take a support agent on an 8B model with grouped-query attention, such as Llama 3 8B: 32 layers, 8 KV heads, head dimension 128, KV stored in BF16. One token's KV is 2 x 32 x 8 x 128 x 2 bytes = 128 KiB. The agent's system prompt and tool schemas are 3,000 tokens, which is 375 MiB of KV. Sixty-four concurrent conversations each add about 200 new tokens per turn.

Prefill work drops by about 92%, and the memory saved goes to a larger batch. In practice the gain depends on hit rate. The tree must still hold the prefix when the next turn arrives, so a burst of unrelated long prompts can evict it. Measure the cache hit rate the server reports (start with --enable-metrics and scrape the Prometheus endpoint) instead of assuming it.

The frontend language

The paper's frontend is a Python embedded language whose calls become requests the runtime can share prefixes across. @sgl.function marks a program, sgl.gen generates, sgl.select picks among options by scoring them, and fork copies the current state into parallel branches:

import sglang as sgl

@sgl.function
def review_ticket(s, ticket):
    s += sgl.system("You triage support tickets.")
    s += sgl.user("Ticket:\n" + ticket)
    branches = s.fork(3)                              # three branches share the prefix above
    for b, aspect in zip(branches, ["urgency", "product area", "sentiment"]):
        b += sgl.user(f"In one line, assess the {aspect}.")
        b += sgl.assistant(sgl.gen("note", max_tokens=48))
    s += sgl.user("Notes: " + " | ".join(b["note"] for b in branches) + "\nRoute it.")
    s += sgl.assistant(sgl.select("queue", choices=["billing", "technical", "account"]))

sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))
state = review_ticket.run(ticket="Charged twice for the same order")
print(state["queue"])

The three branches prefill the ticket once. Most production clients, however, call the OpenAI-compatible /v1/chat/completions endpoint or the native /generate endpoint, and still get prefix reuse, because the cache works on token ids, not on frontend constructs.

Structured output

Constrained decoding masks the logits at each step so that only tokens allowed by a grammar (a JSON schema, regex or EBNF) can be sampled. --grammar-backend selects the implementation: xgrammar, outlines, llguidance or none. The paper adds a compressed finite-state machine. When the grammar allows only one continuation for several tokens, as with a fixed key such as {"queue": ", those tokens can be appended in a single step instead of decoded one at a time.

Two costs to watch: compiling a new schema takes time on first use, so reuse schemas instead of generating a fresh one per request; and a grammar that forces a field before the model has reasoned can lower answer quality. Put free-text reasoning fields before the decision field if you need them.

Launching and tuning

A single-node launch with the settings that most deployments touch:

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --host 0.0.0.0 --port 30000 \
  --tp-size 1 \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 8192 \
  --schedule-policy lpm \
  --enable-metrics

curl -s localhost:30000/generate -H 'Content-Type: application/json' \
  -d '{"text": "Summarise: ...", "sampling_params": {"max_new_tokens": 64, "temperature": 0}}'

• --mem-fraction-static is the share of GPU memory reserved for weights plus the KV pool. Whatever is left covers activations and CUDA graphs. Lower it if you hit out-of-memory errors at startup or under long prompts.
• --tp-size shards one model copy across GPUs. --dp-size runs data-parallel replicas, each with its own scheduler and radix tree.
• --enable-hierarchical-cache lets evicted KV spill to host memory instead of being discarded, which helps when many long conversations come back after a pause.
• --speculative-algorithm enables speculative decoding. Built-ins in current source include EAGLE, EAGLE3 and NGRAM. See speculative decoding.
• POST /flush_cache empties the radix tree. Call it after updating weights in place, because cached KV from old weights is wrong for new ones.

Scaling out without losing the cache

The radix tree is per scheduler, so a cache hit depends on routing. If a round-robin load balancer spreads a conversation's turns over eight replicas, each turn probably lands on a replica that has never seen the prefix, and hit rates collapse. Route with affinity instead: by session id, or by a hash of the leading tokens. The SGLang project ships a router with a cache-aware routing policy for this. Check its documentation for current option names. For very large deployments, separating prefill from decode changes the picture again. See disaggregated serving.

Failure modes

• Prefix poisoning. Inject the current date or a trace id into the opening line of the prompt and no two requests share even their first few tokens, so the hit rate falls to almost nothing. Keep variable content after the stable part.
• Cache thrash. A mix of many distinct long prompts evicts the shared prefixes you care about. Watch the hit rate and evictions together, and give long-document traffic its own replica pool.
• Starvation under lpm. Requests with no cached prefix keep losing the sort. Watch the tail of queue time, not just the mean, and fall back to fcfs if the p99 grows.
• Out of memory after a change. A longer context or a new model shifts the activation peak. Lower --mem-fraction-static or --chunked-prefill-size before you lower the batch.
• Stale KV after weight updates. RL loops that push new weights into a running server must flush the cache, or rollouts are computed partly with old weights.
• Non-determinism. The same prompt at temperature 0 can produce different text depending on batch composition and on whether the prefix came from the cache. Do not build tests that expect byte-identical output from a shared server.

Trade-offs

SGLang's design centre is reuse: agentic, multi-turn, few-shot and structured workloads with shared prefixes. Against vLLM (see vLLM on GPUs), the practical differences are token-granular radix matching against block-hash prefix caching, a different scheduler, and a different set of model and hardware back ends. Both projects move fast and copy good ideas from each other. The paper reports up to 6.4x higher throughput than the systems it compared against, on its own workloads. Treat that as an upper bound for prefix-heavy programs, not a forecast for yours. For workloads with no shared prefix, such as one-off long documents, the radix tree adds bookkeeping and gains nothing, and the engines perform much more alike.

What to do next

1. Measure how much of your traffic shares a prefix: tokenize a day of requests and count the leading tokens they have in common.
2. Move all variable content out of the start of system prompts and tool definitions.
3. Launch one replica with --enable-metrics and replay real traffic. Record the cache hit rate, TTFT and throughput under fcfs and then lpm.
4. Tune --mem-fraction-static and --chunked-prefill-size under your longest real prompts, not under a benchmark.
5. Put session-affinity or cache-aware routing in front before you scale past one replica.
6. Add /flush_cache to any in-place weight-update path, and alert when the hit rate drops sharply.