Why it matters

Response cache is baseline optimization. Understanding shapes design.

Advertisement

The architecture

Compute cache key from prompt + params.

Redis / Caffeine lookup.

Return or call LLM.

Response cache flowKey hashprompt + paramsCache lookupRedis / CaffeineReturn or LLMon hit / missRedis for distributed; Caffeine for in-process; TTL + eviction key
Response cache pattern.
Advertisement

How it works end to end

Caffeine: in-process, high throughput.

Redis: distributed.

Include all params in key.

Deterministic settings preferred.