The three caches in a production LLM system
Application data, repeated prompt prefixes, and final answers behave differently. Treating them as one cache creates stale replies and hides the real cost of an AI product.
Most teams say they have added caching to an LLM product before they have decided what they are actually caching. That sounds like a small detail. It is not. A user profile, a repeated prompt prefix, and a finished assistant reply age at different speeds and fail in different ways.
I find it easier to separate the system into three caches. The first saves work done by your application. The second saves work done by the model provider. The third saves a finished answer. Only the first two are broadly useful.
1. Cache the work your application repeats
Before a request reaches a model, the application often does plenty of work: it loads a workspace, fetches a user profile, reads documents, runs search, builds permissions, and pulls data from outside APIs. That work can be slow and expensive even if the model call itself is cheap.
Redis, a key-value store, or an in-process cache can help here. The key is to cache according to how the data changes.
- A system prompt or tool schema may stay stable for hours or days.
- Search results may be safe for a few minutes.
- A user permission should change as soon as its source changes.
- A third-party API response needs a lifetime based on that API, not a number copied from another cache.
Invalidation matters more than the cache product. If a write changes a record, the application should clear or replace the matching cached value. A long expiry time is not a substitute for knowing when data became wrong.
The largest savings are often inside tools. Search, document loading, large database queries, and paid APIs can cost more than expected when an agent calls them several times in one run. Cache the stable tool result close to the tool. That keeps repeated calls fast without pretending the whole conversation is stable.
2. Structure prompts so provider caching can help
Model providers can reuse work when the beginning of a prompt repeats. This is different from Redis. Your application still sends the request, but the provider can avoid processing the same stable prefix from scratch.
The practical design is simple: stable content first, changing content last. Put the system instructions, long reference material, and stable tool definitions before the user-specific part of the request. If you rebuild or reorder that prefix on every call, there is nothing consistent to reuse.
Do not assume one discount applies everywhere. Cache duration, eligibility, and pricing depend on the provider and model. Check the current documentation and measure actual cache hits. OpenAI documents automatic prompt caching for repeated prefixes, while Anthropic documents cache writes, reads, and their model-specific pricing. Both reward a prompt that has a stable front half.
There is another useful effect. Organising prompts this way makes them easier to inspect. Stable policy is separated from changing user state, which makes a bad request easier to reproduce.
3. Be careful with final-answer caches
A response cache stores the assistant’s entire answer and serves it again when the next request looks similar. It is attractive because the saving appears large: no tools, no retrieval, and no model call.
It is also the easiest cache to get wrong.
The answer may depend on the current user, permissions, conversation history, tool results, time, or a document that changed five minutes ago. A cache key that misses one of those inputs can serve a confident but stale answer. Those bugs are difficult to reproduce because the system appears correct after the entry expires.
Whole-response caching is reasonable for narrow, public, repeatable requests whose inputs can be described completely. It is a poor default for an agent with memory and tools. In most products, caching the expensive work underneath the answer gives most of the benefit with much less risk.
Measure saved work, not cache count
The goal is not to report that the product has three caches. The useful numbers are the time and money each layer saves, plus the amount of stale data it serves.
Track cache hit rate, but also track what a hit avoided. A hit on a tiny user record is not equal to a hit on a paid search call. Record age and invalidation reason as well. If nobody can explain why an entry is still valid, it should not be trusted.
My rule is straightforward: cache stable inputs and expensive intermediate work. Recompute changing state. Treat a final answer as fresh unless you can prove every input that shaped it is represented in the cache key.