When does prompt caching actually save money?

Prompt caching is the highest-leverage cost optimisation available to most LLM applications, and it is routinely implemented in a way that never hits. The mechanism is simple; the failure modes are all about prompt layout.

What is being cached, exactly?

A prefix, matched exactly from the first token.

Both Anthropic and OpenAI implement this the same way, with different pricing and expiry rules.

When you mark a cache breakpoint, the provider stores the processed representation of everything before it. A later request whose prompt begins with those identical tokens skips the reprocessing and reads the stored state instead.

Token cost relative to one base input token

Show data

Token cost relative to one base input token. Bar chart. Cache write: 1.25×. Base input token: 1×. Cache read: 0.1×.

Item	Value (×)	Note
Cache write	1.25	Charged once, when the prefix is first stored.
Base input token	1
Cache read	0.1	Charged on every subsequent hit within the TTL.

Two consequences follow directly, and they explain almost every disappointing result:

• The match is byte-exact and anchored at the start. Nothing about it is semantic or fuzzy.
• One changed character early in the prompt invalidates everything after it.

One thing to be precise about. “Processed representation” means the provider stores the key/value attention state for those tokens, the KV cache, rather than the text. The work being skipped is the expensive part of inference, which is where the saving comes from. That state also occupies memory on serving hardware, so providers attach a short time-to-live, commonly a few minutes, refreshed on each hit.

Both facts point the same way. Caching rewards traffic that reuses a large prefix quickly, and does nothing for traffic that does not.

How do you lay out a cacheable prompt?

Order the prompt by how often each part changes, most stable first:

Prompt layout, ordered most stable to most volatile

Show as text

Prompt layout, ordered most stable to most volatile. Diagram: 7 ordered layers. System instructions and role definition, then Tool and function definitions, then Large reference documents, schemas, examples, then cache breakpoint (breakpoint), then Conversation history, then Current user input, then Timestamps, request IDs, retrieved passages.

#	Layer	Note
1	System instructions and role definition	Changes on deploy, if ever
2	Tool and function definitions	Changes on deploy
3	Large reference documents, schemas, examples	The bulk of the cached prefix
·	cache breakpoint (breakpoint)	everything above is the cache key
4	Conversation history	Grows per turn; a 2nd breakpoint can follow
5	Current user input
6	Timestamps, request IDs, retrieved passages	Must never appear above the breakpoint

1. System instructions and role definition
2. Tool and function definitions (an MCP toolset lands here)
3. Large reference documents, schemas, examples
4. ← cache breakpoint
5. Conversation history (or a second breakpoint that advances with the turns)
6. The current user input
7. Anything genuinely per-request: timestamps, IDs, retrieved passages

The classic mistake is injecting a current date or a request ID into the system prompt “for context”. It costs nothing to read and guarantees a total miss on every request.

The second most common is non-deterministic serialisation. If your tool schema is assembled from a dictionary and serialised without sorting keys, byte order can differ between processes or language versions, and the prefix silently stops matching on some machines but not others. It shows up as a hit rate stuck around 60% for no visible reason. Sort the keys, pin the ordering, and treat the cached prefix as a build artefact rather than something reassembled per request.

// Stable-to-volatile ordering. Everything before the breakpoint is the
// cache key, so a single varying character above it costs you every hit.
const messages = [
  { role: "system", content: [
      { type: "text", text: SYSTEM_INSTRUCTIONS },        // never changes
      { type: "text", text: JSON.stringify(TOOL_SCHEMA) },// changes on deploy
      { type: "text", text: REFERENCE_DOC,
        cache_control: { type: "ephemeral" } },           // <- breakpoint
  ]},
  ...history,                                             // grows per turn
  { role: "user", content: userInput },                   // varies every call
];
// WRONG, and the most common bug in the wild:
//   text: `Today is ${new Date().toISOString()}. ${SYSTEM_INSTRUCTIONS}`

When does prompt caching break even?

Let a base input token cost 1 unit. A cache write costs roughly 1.25 units, a cache read roughly 0.1. For a stable prefix of N tokens reused across R requests:

• Without caching: N × R units
• With caching: N × 1.25 for the first write, then N × 0.1 × (R − 1)

Cumulative cost of a 20,000-token prefix

• No caching
• With caching

Show data

Cumulative cost of a 20,000-token prefix. Line chart of Cumulative cost against Requests within the cache TTL. No caching ranges from 1 to 50. With caching ranges from 1.25 to 6.15.

Requests within the cache TTL	No caching	With caching
1	1	1.25
2	2	1.35
3	3	1.45
5	5	1.65
10	10	2.15
20	20	3.15
50	50	6.15

Setting them equal, caching wins once R is above roughly 1.3. The second request inside the TTL already pays for the write. Verify the exact multipliers against your provider’s current pricing, but the shape holds: the break-even is low enough that the real question is only whether your traffic reuses prefixes inside the expiry window.

The TTL is what actually decides this, far more than the multipliers. A five-minute window turns caching into a bet on request density: bursty traffic hits it, and evenly spread traffic quietly pays the write premium again and again. Some providers sell a longer window at a higher write price, worth buying precisely when your reuse is real but slow.

Workload	Stable prefix	Reuse in TTL	Caching
Chat with long system prompt	Large	Every turn	Strongly positive
Agent loop over a fixed toolset	Large	Every step	Strongly positive
Document Q&A, many questions per doc	Very large	High	Strongly positive
One-shot classification, unique inputs	Small	None	Net loss
Batch job, distinct document each time	None	None	Net loss

How do you confirm it is working?

Do not infer it from the bill. Both providers return per-request token counts that separate cache writes from cache reads, documented in the Anthropic usage fields and the equivalent OpenAI response object. Log those two numbers as a ratio and alert when the read share drops. A deploy that reorders tool definitions or adds a line to the system prompt will silently zero your hit rate, and the only visible symptom is a cost increase nobody attributes to the right change.

# Emit this per request. Alert on a sustained drop rather than one miss:
# a deploy that changes the prefix shows up here long before it shows up
# on the invoice.
def cache_health(usage) -> dict:
    read = usage.cache_read_input_tokens
    write = usage.cache_creation_input_tokens
    total = read + write + usage.input_tokens
    return {
        "hit_ratio": read / total if total else 0.0,
        "write_ratio": write / total if total else 0.0,
        "uncached": usage.input_tokens,
    }

Two things make that alert trustworthy. Compare against a rolling baseline rather than a fixed threshold, because a healthy hit rate differs per endpoint. A number that looks fine for a one-shot classifier is alarming for an agent loop. And bucket the metric by deploy version, so a regression points at the change that caused it rather than at the hour somebody noticed.

Treat misses as a correctness smell as well as a cost one. A prefix that stops matching almost always means something varied that you believed was constant, and that is worth understanding even when the money involved is trivial.