Why it matters

Long conversations need efficient KV management. StreamingLLM enables.

Advertisement

The architecture

Keep first few tokens (attention sinks) forever.

Roll window of recent tokens.

Drop middle.

StreamingLLM approachAttention sinksfirst tokens fixedRolling windowrecent tokensDrop middleold contextOnly works because attention concentrates on initial + recent tokens
StreamingLLM strategy.
Advertisement

How it works end to end

Discovery: attention concentrates on initial tokens (sinks).

Rolling window: keep last N tokens.

Enables infinite context length in principle.