What is prompt caching? Explained with a chef's prepped base sauce
Prompt caching lets an AI model reuse its work on the repeated start of a prompt, so that part costs less and replies start sooner. Explained with a chef who keeps a prepared base sauce on the shelf.
- Author
Umair Mateen- Published
- Reading time
- 5 min
- Category
- AI
Two AI price lists we covered this week, for Meta's Muse Ember and Alibaba's Qwen4-Max, show a separate, much lower price for "cached input" or "cache read". That price comes from prompt caching. Here is what it means.
In one sentence: Prompt caching lets an AI model reuse the work it already did reading the start of a prompt, when the next request begins with exactly the same text, so that repeated part costs less and the answer can start sooner.
The everyday picture
Picture a busy restaurant kitchen. Many dishes start from the same base sauce. Making that base takes time: chopping, frying, simmering. Only the last few steps differ from dish to dish.
A smart chef makes the base once and keeps it on a prep shelf. When the next order starts with that same base, the chef takes it from the shelf and only cooks the new part. The base is not thrown away after one dish, but it does not keep forever either. If nobody uses it for a while, it is cleared away.
There is one strict rule. The shelf only helps if the order starts with the exact same base. Change one ingredient at the beginning, and the chef has to start from scratch.
When a prompt starts exactly like a recent one, the provider reuses the processed start from its cache. The model then only reads the new part and writes a fresh answer.
How it works, step by step
- A request arrives (the order). It has a long fixed part, such as instructions, a product manual or tool descriptions, followed by a short new part, such as a customer's question.
- The provider checks the start of the prompt (does the order begin with a base we already made?). It compares the opening part, called the prefix, with prompts it has processed recently. It must match exactly, word for word.
- If there is no match, the model reads everything (the chef makes the base from scratch). The provider then keeps the processed start on its cache (the prep shelf). Some providers charge a little extra for this first write; Anthropic, for example, charges 1.25 times the normal input price for its default five-minute cache.
- If there is a match, the model reuses that work (the chef takes the base from the shelf). It only has to read the new part. Anthropic bills these cached tokens at a tenth of the normal input price, or less on some models.
- The cache expires after a while (old prep is cleared away). Anthropic's default lifetime is five minutes, renewed each time the cache is used. Google's explicit cache, on models that support it, defaults to one hour and charges for storage time.
Try it · What would you do?
Which prompt layout gets the cached price?
A support assistant sends the same 40-page product manual with every customer question. The developer also adds the current time and the customer's name to each request. Which layout lets prompt caching help most?
Pick what you would do. You can change your mind.
Where the picture stops
- A kitchen can reuse a base sauce in any dish. A prompt cache only works for the start of a prompt. Matching text in the middle or at the end does not count.
- The shelf holds the model's processed reading of your text, not a finished answer. The model still writes a fresh answer every time. Anthropic's docs say caching does not change the response you get.
- A real chef decides what to keep. With AI, the provider's rules decide: some cache automatically, others only when your developer marks the part to keep. Prompts that are too short are not cached at all.
Why it matters for your business
- Lower running costs. Many AI tools send the same long instructions or documents with every request. Caching that part can cut the input bill for it sharply.
- Faster replies. Anthropic says time to the first word of the answer generally improves for long documents, because the model skips re-reading them.
- Prompt design affects the bill. Providers advise putting fixed content first and changing details, such as today's date or the user's name, last. A small change at the top can cancel the saving.
- Read price lists carefully. A low "cached input" price only applies when your prompts actually repeat. Compare models using your own traffic, not the best-case price.
An example
Imagine a 30-person insurance broker with an AI assistant that answers staff questions about policy wording. Every request sends a 40-page policy guide, then the question. At first the developer put the date and the staff member's name at the very top, so the start of every prompt was different and nothing was reused. After moving the guide to the top and the name, date and question to the end, back-to-back questions within a few minutes reuse the cached guide. The answers stay the same; the repeated part simply costs less.
Frequently asked questions
Does prompt caching make AI answers less accurate?
No. The cache stores the model's processed reading of the prompt, not a saved answer. Anthropic's documentation says the response is the same as it would be without caching.
Is prompt caching the same as AI memory?
No. The model does not remember earlier conversations because of it. You still send the full prompt every time; the provider just avoids re-processing the part it has seen recently.
Do I have to turn prompt caching on?
It depends on the provider and model. Google says implicit caching is on automatically for Gemini 2.5 and newer models, and OpenAI applies caching automatically on supported models. With Anthropic, your developer marks what to cache, or turns on its automatic mode.
How long does a prompt cache last?
Usually minutes, not days. Anthropic's default is five minutes, refreshed each time it is used, with a one-hour option at a higher write price. Google's explicit cache defaults to one hour and can be set longer or shorter.
Related explainers
The DevDrift builds AI features designed to keep running costs predictable: see how.
Sources
Could you explain this to a colleague now?
Mark it and posts that mention it will show a tick. Saved in this browser only.