# Prompt Caching (Part 1): What Is Actually Cached > Prompt caching saves the server's prefill compute, not the cost of re-sending tokens. Where the KV cache comes from, why a prefix must match token for token, and why caching can cost more. - Author: zhuermu - Published: 2026-08-17 - Web version: https://zhuermu.com/en/blog/prompt-cache-1-how-it-works/ --- A claim that circulates widely: prompt caching saves you the cost of re-sending repeated tokens. It is wrong, and it is wrong in a way that has consequences — it makes caching sound like a network optimisation, so you go looking for the win in the wrong place. The prompt is transmitted in full every time, to the byte. What you stop paying for is **the server computing it again**. Once that lands, every counter-intuitive thing about this feature has an explanation: why editing a single space voids everything, why switching caching on can cost you *more*, why the same model on a different cloud has a different cost model. This is part one, and it is only about the mechanism. [Part two](/blog/prompt-cache-2-platform-comparison/) puts five model vendors and three clouds side by side at current prices. [Part three](/blog/prompt-cache-3-debugging-cache-misses/) covers the engineering mistakes that actually destroy a cache in production. [Part four](/blog/prompt-cache-4-bedrock-threshold-test/) is a measured run on Bedrock that turned up an error in the vendor documentation. ## What is cached is prefill, not traffic Start with what the server does with one request. ![Prefill produces the KV cache; a cache hit skips that whole phase and pays roughly a tenth for the read](/images/blog/prompt-cache/01-prefill-decode.svg) When a transformer generates, every token attends to all the tokens before it. The K/V projections of those earlier tokens do not need recomputing once they exist, so they are kept — that is the **KV cache**. A request has two phases, and their characteristics are nothing alike. **Prefill** processes the entire prompt in one parallel forward pass. It is **compute-bound**. On a 100k-token context this phase is a genuinely large pile of floating-point work. **Decode** emits output one token at a time, and each step reads back that KV state. It is **memory-bandwidth-bound**. What `prompt caching` does is keep the KV state that prefill produced, across requests. Send the same prefix again, the server recognises it, skips prefill and goes straight to decode. So the benefit is twofold — **cost** (a read is billed at roughly 10% of the input rate) and **time to first token** (that pile of prefill compute is not spent again). In agent workloads the second one is often worth more than the first. One corollary that is easy to miss: **caching has no effect on the output.** The documentation is explicit — the model regenerates from the cached prefix and sampling is unchanged, so two identical requests differ exactly as much as they would have anyway. It saves repeated computation; it does not memoise answers. That is a different thing from the "response cache" most people picture first. ## Why the prefix has to match token for token If what is cached is KV state, can you change one word and reuse the rest? No. And the reason is in the definition of attention. ![Change the system block and everything after it is voided, recomputed, and rebilled at write price](/images/blog/prompt-cache/01-prefix-hash.svg) Attention is **causal** — the K/V at position k depends on every token before it. Change token 5 and the KV of every token after 5 is wrong. So a cache can only be matched by **prefix**, and only by a prefix that is identical token for token. Servers typically hash the prefix in blocks (64 tokens is a common size) and skip the span that matches. Which produces the one iron rule in this whole subject: > **Stable content first, volatile content last.** System prompt, tool definitions and reference documents in front; timestamps, session IDs and this turn's user input behind. A dynamic timestamp at the top of a prompt zeroes the entire cache. **And it does not raise an error** — inference still succeeds, the answer is still correct, only the bill goes up. That is what makes this class of bug so unpleasant, and it is what part three is about. All three major vendors call this scenario out in their documentation. OpenAI's advice is the practical one: a dynamic value you only need for logging or debugging belongs in request metadata, not in the prompt. ### The order of the layers is part of the prefix Anthropic defines the prefix hierarchy explicitly: `tools` → `system` → `messages`. Change an earlier layer and the cache for every layer after it goes with it. Rename a tool, reword its description, or adjust a parameter, and **all three layers are void**. OpenAI puts it even more bluntly: tool definitions, **the order of the tools**, and the schema for structured output all participate in forming the prefix. That "the order counts too" clause is the one real systems trip over. If your tool list is collected out of a map, or registered dynamically by plugins, the order can differ between process restarts. Part three has a reproducible example. ## Why a TTL has to exist, and why it slides KV state is large. 100k tokens on a frontier model is tens of gigabytes, sitting in HBM or on SSD. No vendor is going to hold that for you indefinitely, so it has to be evicted. Hence a TTL. And two details about it decide how you actually tune anything. ![A read refreshes the TTL for free; an interval longer than the TTL turns the optimisation into the problem](/images/blog/prompt-cache/01-ttl-window.svg) **First, the TTL refreshes on read, and the refresh is free.** Anthropic's documentation states it plainly: the cache is refreshed every time it is used, at no additional charge. This is a sliding window, not a fixed countdown from the write. The entire idea of a keepalive rests on that one sentence — during a pause, resend the same prefix on a timer and the window keeps extending. And this is not a community hack: Anthropic's own documentation recommends a warm-up request at least every five minutes for the five-minute tier, and provides a request shape for it, `max_tokens: 0`, which reads the prompt, writes the cache, returns immediately, and **is not billed for output tokens**. **Second, once the interval exceeds the TTL, this optimisation changes sign.** This is the part I think is most worth remembering. Suppose the TTL is five minutes and you ping every eight. Every single ping now lands on a cache that is already dead. So it is not a cheap read any more — it is a **full-price prefill, plus a write fee**. An action meant to cost 0.1× costs 1.25× instead: 12.5× more per call, and you are doing it on a loop. Maxim Khailo measured exactly this cell in his [cross-vendor keepalive tests](https://blog.mempko.com/your-agentic-workflows-cache-keepalive-costs-8x-too-much-v2-the-interval-frontier/) — on Anthropic, an eight-minute interval billed **4× what not pinging at all would have cost**. Most parameters, tuned badly, just work less well. This one has a cliff. **Past the line, the optimisation becomes the pathology.** ### A timing trap that is easy to read past There is one more sentence in Anthropic's documentation, and I missed it the first time: **The lifetime is counted from the start of the request that wrote or read the cache, not from the end of the response. Time spent generating counts against the TTL.** An example makes the severity obvious: if a streamed response takes four minutes, the next request reusing that prefix has to go out within roughly one minute of it finishing. Which means the "four minutes is optimal" figure circulating in the community is **a conclusion drawn at idle**. In a real agent workload, a long reply with extended thinking streaming for two or three minutes is entirely ordinary, and all of that time is inside the five. **Your safe interval has to be shorter than four minutes, and how much shorter depends on the distribution of your own response times.** Worth noting in passing: Aider was the first tool to ship cache keepalive publicly (`--cache-keepalive-pings`, from v0.53.0), and the interval it uses is **five minutes** — sitting exactly on Anthropic's TTL boundary. By the rule above, that interval risks overshooting whenever responses stream for a while. It is a thing you can measure for yourself. ## Writes are not free, so caching can make you poorer The last piece: writing the cache costs money. - **Anthropic** — write at **1.25×** the base input rate for the five-minute tier, **2×** for the one-hour tier, read at **0.1×** - **OpenAI** — writes were free before GPT-5.6; **from GPT-5.6 the write costs 1.25×** - **Claude on Bedrock** — the same 1.25× / 2× / 0.1× as Anthropic's first-party API - **Gemini** — reads are a flat 90% off (so 0.1×), but explicit caching **also bills storage by token-hour** Put those together and one conclusion surfaces: > **If your prefix changes on every call, caching costs more than not caching.** Because you pay the write premium every time and never collect a read. This is not hypothetical — the "a proxy silently dropped the cache markers" cases in part three end up with exactly this bill shape: the feature works, the logs are clean, the cost goes up. There is a subtler threshold too: **a prefix that is too short is silently not cached.** Every vendor has a minimum cacheable token count, and the behaviour below it is that inference succeeds, nothing is cached, and **nothing is reported**. Anthropic documents thresholds ranging from 512 to 4,096 tokens depending on the model. Those numbers are not necessarily reliable either. I measured Claude Sonnet 4.5 on Bedrock and **the real threshold is 1,024 where the AWS documentation says 4,096** — off by 4×. Three other models measured with the same harness agreed with their documented figures. So a threshold is worth verifying once before you depend on it; two requests settle it, and the method is in [part four](/blog/prompt-cache-4-bedrock-threshold-test/). If you want to know which range is even worth probing, measure your real system prompt with the [token counter](/tools/token-counter/) first. ## In short Four points, in order of how much they matter. **What is cached is the server's prefill compute, not network traffic.** The benefit is cost plus time-to-first-token, and it does not change what the model says. **The prefix must be identical token for token, and everything after an edit is voided with it.** From which follows the single rule: stable first, volatile last. Tool definitions — and tool order — count as prefix. **The TTL is a sliding window and reads refresh it for free, but the clock starts at the request, so streaming time counts.** An interval longer than the TTL inverts the optimisation into a 4× cost. **Writes carry a premium, so an unstable prefix is a straight loss.** And a prefix that is too short is silently not cached — the threshold varies by model, and the documented number may be wrong. [Part two](/blog/prompt-cache-2-platform-comparison/) lays out the current multipliers, TTL tiers, minimum thresholds and cross-region routing behaviour for five model vendors and three clouds, and works out which knob is worth turning on which platform, at which pause length. Written in mid-August 2026. Every price and multiplier here comes from the vendor documentation current at that date, cited at the end. Pricing in this area has an alarmingly short half-life — one of these vendors changed its billing model on the very day I was compiling the figures. Verify before you copy a conclusion. --- ## Frequently asked ### Does prompt caching save network bandwidth? No. The prompt is sent to the server in full on every call, to the byte, and nothing about transfer changes. What you stop paying for is the server's prefill — the KV state it just computed is reused, so the read is billed at roughly a tenth of the input rate, and most of the time-to-first-token disappears with it. ### Why does changing one word invalidate the whole cache? Attention is causal: the K/V of every token depends on all the tokens before it. Edit token k and every KV after k is wrong. So a cache can only be matched by prefix, and the prefix has to be identical token for token. ### How can a cache write cost more than not caching? Most vendors charge a premium to write: Anthropic bills 1.25× for the five-minute tier and 2× for the one-hour tier, and OpenAI charges 1.25× from GPT-5.6 onward. If your prefix changes on every call you pay that premium every time and never get a read, which is strictly worse than leaving caching off. ### Is there a minimum prompt length for caching? Yes, and falling short of it fails silently — inference succeeds, nothing is cached, and no error is raised. Anthropic documents thresholds from 512 to 4,096 tokens depending on the model. Those numbers are worth verifying yourself: on Bedrock I measured Claude Sonnet 4.5 caching from 1,024 tokens while AWS documents 4,096. --- ## References - [Prompt caching](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — Anthropic documentation - [Prompt caching](https://platform.openai.com/docs/guides/prompt-caching) — OpenAI documentation - [Prompt caching for faster model inference](https://docs.aws.amazon.com/bedrock/latest/userguide/prompt-caching.html) — AWS Bedrock documentation - [Context caching](https://ai.google.dev/gemini-api/docs/caching) — Google Gemini API documentation - [Your Agentic Workflow's Cache Keepalive Costs 8x Too Much (v2: the interval frontier)](https://blog.mempko.com/your-agentic-workflows-cache-keepalive-costs-8x-too-much-v2-the-interval-frontier/) — Maxim Khailo — cross-vendor keepalive measurements