---
title: When does prompt caching actually save money?
url: https://deepthinkingai.org/prompt-caching-economics/
published: 2026-09-11
author: Shekhar Singh
topic: AI Engineering
tags: prompt-caching, cost, latency, production
site: DeepThinking AI
---
# When does prompt caching actually save money?
**Summary:** Prompt caching stores a prefix of your prompt so later requests reuse it instead of reprocessing it. Reads are far cheaper than base input tokens, but writing the cache costs a premium and entries expire. It pays whenever a large stable prefix is reused several times inside the TTL, and loses on one-shot traffic.
## Key takeaways
- Caching applies to a prompt prefix, so anything that varies must go after everything stable.
- A cache write costs more than a normal input token; a read costs a fraction of one.
- Break-even arrives after a small number of reads within the TTL, often two or three.
- A single changed character early in the prompt invalidates the whole prefix, which is the most common cause of silent cache misses.
Prompt caching is the highest-leverage cost optimisation available to most LLM
applications, and it is routinely implemented in a way that never hits. The
mechanism is simple; the failure modes are all about prompt layout.
## What is being cached, exactly?
A **prefix**, matched exactly from the first token.
Both [Anthropic](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)
and [OpenAI](https://platform.openai.com/docs/guides/prompt-caching) implement
this the same way, with different pricing and expiry rules.
When you mark a cache breakpoint, the provider stores the processed
representation of everything before it. A later request whose prompt begins with
those identical tokens skips the reprocessing and reads the stored state instead.
**Token cost relative to one base input token**
| Item | Value (×) | Note |
|---|---|---|
| Cache write | 1.25 | Charged once, when the prefix is first stored. |
| Base input token | 1 | |
| Cache read | 0.1 | Charged on every subsequent hit within the TTL. |
Illustrative multipliers in the range providers commonly publish. Check your provider's current pricing before modelling on these.
Two consequences follow directly, and they explain almost every disappointing
result:
- The match is byte-exact and anchored at the start. Nothing about it is semantic or fuzzy.
- One changed character early in the prompt invalidates everything after it.
One thing to be precise about. "Processed representation" means the provider
stores the key/value attention state for those tokens, the KV cache, rather than
the text. The work being skipped is the expensive part of inference, which is
where the saving comes from. That state also occupies memory on serving hardware,
so providers attach a short time-to-live, commonly a few minutes, refreshed on
each hit.
Both facts point the same way. Caching rewards traffic that reuses a large prefix
quickly, and does nothing for traffic that does not.
## How do you lay out a cacheable prompt?
Order the prompt by how often each part changes, most stable first:
**Prompt layout, ordered most stable to most volatile**
1. System instructions and role definition
Changes on deploy, if ever
2. Tool and function definitions
Changes on deploy
3. Large reference documents, schemas, examples
The bulk of the cached prefix
--- cache breakpoint --- (everything above is the cache key)
4. Conversation history
Grows per turn; a 2nd breakpoint can follow
5. Current user input
6. Timestamps, request IDs, retrieved passages
Must never appear above the breakpoint
A single varying character above the breakpoint invalidates the whole prefix, which is why per-request values belong at the bottom.
1. System instructions and role definition
2. Tool and function definitions (an [MCP](/how-model-context-protocol-works/) toolset lands here)
3. Large reference documents, schemas, examples
4. **← cache breakpoint**
5. Conversation history (or a second breakpoint that advances with the turns)
6. The current user input
7. Anything genuinely per-request: timestamps, IDs, retrieved passages
The classic mistake is injecting a current date or a request ID into the system
prompt "for context". It costs nothing to read and guarantees a total miss on
every request.
The second most common is non-deterministic serialisation. If your tool schema
is assembled from a dictionary and serialised without sorting keys, byte order
can differ between processes or language versions, and the prefix silently stops
matching on some machines but not others. It shows up as a hit rate stuck around
60% for no visible reason. Sort the keys, pin the ordering, and treat the cached
prefix as a build artefact rather than something reassembled per request.
```ts
// Stable-to-volatile ordering. Everything before the breakpoint is the
// cache key, so a single varying character above it costs you every hit.
const messages = [
{ role: "system", content: [
{ type: "text", text: SYSTEM_INSTRUCTIONS }, // never changes
{ type: "text", text: JSON.stringify(TOOL_SCHEMA) },// changes on deploy
{ type: "text", text: REFERENCE_DOC,
cache_control: { type: "ephemeral" } }, // <- breakpoint
]},
...history, // grows per turn
{ role: "user", content: userInput }, // varies every call
];
// WRONG, and the most common bug in the wild:
// text: `Today is ${new Date().toISOString()}. ${SYSTEM_INSTRUCTIONS}`
```
## When does prompt caching break even?
Let a base input token cost 1 unit. A cache write costs roughly 1.25 units, a
cache read roughly 0.1. For a stable prefix of *N* tokens reused across *R*
requests:
- **Without caching:** `N × R` units
- **With caching:** `N × 1.25` for the first write, then `N × 0.1 × (R − 1)`
**Cumulative cost of a 20,000-token prefix**
| Requests within the cache TTL | No caching | With caching |
|---|---|---|
| 1 | 1 | 1.25 |
| 2 | 2 | 1.35 |
| 3 | 3 | 1.45 |
| 5 | 5 | 1.65 |
| 10 | 10 | 2.15 |
| 20 | 20 | 3.15 |
| 50 | 50 | 6.15 |
The lines cross between the first and second request. Past roughly two reads inside the TTL, caching is strictly cheaper and the gap widens without bound.
Setting them equal, caching wins once *R* is above roughly 1.3. The **second**
request inside the TTL already pays for the write. Verify the exact
multipliers against your provider's current pricing, but the shape holds: the
break-even is low enough that the real question is only whether your traffic
reuses prefixes inside the expiry window.
The TTL is what actually decides this, far more than the multipliers. A five-minute
window turns caching into a bet on request density: bursty traffic hits it, and
evenly spread traffic quietly pays the write premium again and again. Some providers sell a
longer window at a higher write price, worth buying precisely when your reuse is
real but slow.
| Workload | Stable prefix | Reuse in TTL | Caching |
|---|---|---|---|
| Chat with long system prompt | Large | Every turn | Strongly positive |
| Agent loop over a fixed toolset | Large | Every step | Strongly positive |
| [Document Q&A](/long-context-vs-retrieval/), many questions per doc | Very large | High | Strongly positive |
| One-shot classification, unique inputs | Small | None | Net loss |
| Batch job, distinct document each time | None | None | Net loss |
## How do you confirm it is working?
Do not infer it from the bill. Both providers return per-request token counts
that separate cache writes from cache reads, documented in the
[Anthropic usage fields](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching)
and the equivalent OpenAI response object. Log those two numbers as a ratio and
alert when the read share drops. A deploy that reorders tool definitions or adds a
line to the system prompt will silently zero your hit rate, and the only visible
symptom is a cost increase nobody attributes to the right change.
```python
# Emit this per request. Alert on a sustained drop rather than one miss:
# a deploy that changes the prefix shows up here long before it shows up
# on the invoice.
def cache_health(usage) -> dict:
read = usage.cache_read_input_tokens
write = usage.cache_creation_input_tokens
total = read + write + usage.input_tokens
return {
"hit_ratio": read / total if total else 0.0,
"write_ratio": write / total if total else 0.0,
"uncached": usage.input_tokens,
}
```
Two things make that alert trustworthy. Compare against a rolling baseline
rather than a fixed threshold, because a healthy hit rate differs per endpoint.
A number that looks fine for a one-shot classifier is alarming for an agent
loop.
And bucket the metric by deploy version, so a regression points at the change
that caused it rather than at the hour somebody noticed.
Treat misses as a correctness smell as well as a cost one. A prefix that stops
matching almost always means something varied that you believed was constant,
and that is worth understanding even when the money involved is trivial.
## Turn on prompt caching and confirm it is actually hitting
Most caching work fails silently, so the ordering steps and the measurement steps matter equally.
1. **Sort your prompt blocks from never-changing to per-request**: System instructions first, then tool definitions, then large reference material. Conversation history, user input, timestamps and retrieved passages all go after.
2. **Serialise every block deterministically**: Sort object keys and pin array ordering before rendering. Non-deterministic JSON is the reason a hit rate sits near 60% with no obvious cause.
3. **Place the breakpoint at the last byte that never varies**: Everything above it becomes the cache key. Moving it even one block too low throws away most of the saving.
4. **Remove anything dynamic from above the breakpoint**: A current date in the system prompt is the classic case. It reads as harmless and guarantees a total miss on every request.
5. **Log cache reads, cache writes and uncached input per request**: Providers return all three. Track the read share as a ratio and bucket it by deploy version so a regression names the change that caused it.
6. **Alert on a sustained drop against a rolling baseline**: A fixed threshold will be wrong for at least one endpoint. Compare each route against its own recent history instead.
## Frequently asked questions
### What actually gets cached?
A contiguous prefix of the prompt, matched exactly from the beginning. It is not a semantic or fuzzy cache. The tokens must match byte for byte from the start of the prompt up to the cache breakpoint.
### Why is my cache hit rate near zero?
Almost always because something variable appears early in the prompt: a timestamp, a request ID, a session identifier or a reordered tool list. Anything that changes per request has to sit after all cached content.
### Does caching change the model's output?
No. It changes how the input is processed. What the model computes is unchanged. Identical inputs produce the same distribution over outputs whether or not the prefix was cached.
### Is it worth it for a chatbot?
Usually yes, because the system prompt and conversation history form a growing stable prefix reused on every turn. It is single-shot classification and batch jobs with unique inputs where caching is a net loss.
## Sources
- [Prompt caching](https://docs.anthropic.com/en/docs/build-with-claude/prompt-caching). Anthropic
- [Prompt caching for faster model inference](https://platform.openai.com/docs/guides/prompt-caching). OpenAI
---
Canonical HTML: https://deepthinkingai.org/prompt-caching-economics/