# Prompt Caching (Part 3): It Is On and Not Hitting — Now What

> Cache invalidation almost never errors: inference succeeds, the logs are clean, only the bill grows. Four culprits by frequency, plus a cross-vendor hit-rate self-check you can run today.

- Author: zhuermu
- Published: 2026-08-17
- Web version: https://zhuermu.com/en/blog/prompt-cache-3-debugging-cache-misses/

---
**Verdict first.**

Prompt cache invalidation **almost never raises an error**. Inference succeeds, the answer is correct, the logs are clean, and the bill goes up. That is what makes it hard to find, and it is the reason this is one of the few things you have to monitor deliberately rather than trust.

Ordered by how often I have actually seen them, the culprits fall into four groups:

1. **Volatile content inside the prefix** — timestamps, session IDs, context compaction. The most common, and the easiest to fix.
2. **Serialisation order jitter** — the same data structure producing different bytes on two passes.
3. **Unstable tool ordering** — `tools` is part of the prefix, and its order is often decided by the filesystem or the thread scheduler.
4. **A middle layer silently eating the cache markers** — you set `cache_control`, a proxy removed it. The nastiest of the four.

If you already have a service you suspect is not hitting, skip to the self-check scripts at the end; five minutes gets you an answer.

[Part one](/blog/prompt-cache-1-how-it-works/) covered the mechanism and [part two](/blog/prompt-cache-2-platform-comparison/) the platform differences. This part is the debugging.

## Calibrate the terminology first

If your instinct comes from the Redis-and-MySQL world, it is worth separating two things that sound alike:

| Term | Meaning | Where it applies |
|---|---|---|
| **cache miss / invalidation** | the prefix hash does not match, so nothing is reused: full-price recompute plus a rewrite | LLM prompt caching |
| **cache penetration** | a query for a key that **does not exist anywhere**, passing through the cache to the database | Redis + MySQL style architectures |

Prompt caching has no notion of a non-existent key — the prefix is either there and hit, or absent and recomputed. Calling it invalidation, or just a miss, keeps the investigation pointed in the right direction.

## Culprit 1: volatile content inside the prefix

This accounts for most of what I have seen, and the reason is the plainest: **attention is causal, so every KV after an edit is void.** Anything volatile sitting near the front zeroes the whole cache.

All three vendors name specific cases in their documentation.

**A dynamic timestamp at the top of the prompt.** The classic. `"Current time: 2026-08-17 10:25:07"` at the head of a system prompt changes every second, and the hit rate is identically zero.

**A session ID or user ID in the system prompt.** Different for every conversation, so every conversation writes the cache afresh and never reads it. OpenAI's advice here is the practical one: **a dynamic value you only need for logging or debugging belongs in request metadata, not in the prompt.**

**Context compaction resets the prefix.** This one gets missed because compaction is itself a good optimisation. OpenAI's documentation is clear about it: truncation, summarisation and compaction reduce prompt size but also reset the reusable prefix. **Two optimisations fighting each other** — the tokens you save by compacting may be fewer than the tokens you now pay full price for. That trade-off has to be computed on your own data; there is no general answer.

**Rewriting or reordering history.** Multi-turn conversations should append, not edit. Delete an early message, or reorder the history, and the prefix has changed.

Beyond those, there is a whole class of "parameters that turn out to be prefix", all of them documented:

- image order, or a change to the `detail` parameter
- a change to the structured-output schema
- a change to the thinking configuration or `budget_tokens`
- a change to `output_config.effort`
- a change to `tool_choice`
- toggling web search or citations, which rewrite the system prompt
- switching between fast mode and standard speed

What those last few have in common: **they look like request parameters and are actually rendered into the prompt.** An A/B experiment that varies the thinking budget will miss the cache for its entire duration.

## Culprit 2: serialisation order jitter — and one vendor misattribution

Anthropic's troubleshooting section carries a warning to the effect that you should make sure the key order inside `tool_use` content blocks is stable, because some languages — Swift and Go are named — randomise key order during JSON conversion and break the cache.

The direction is right. **Naming languages is not.** I went and checked Go's own documentation.

The `encoding/json` page on `pkg.go.dev` is explicit: map keys are **sorted** before being used as JSON object keys. The v1/v2 differences section goes further — in v1, Go maps serialise in a **deterministic** order, while v2 is non-deterministic and controlled by the `jsonv2.Deterministic` option.

So the v1 standard library is precisely the safe case. The real risk distribution looks like this:

```
json.Marshal(someMap)             ✅ keys already sorted, output stable
json.Marshal(someStruct)          ✅ declaration order, output stable
hand-rolled for k, v := range m   ❌ the Go spec guarantees no map iteration order
encoding/json v2                  ❌ non-deterministic by default, needs Deterministic
Swift JSONEncoder                 ❌ unsorted by default (.sortedKeys is opt-in)
Python json.dumps                 ⚠️ sort_keys defaults to False, but 3.7+ dicts keep
                                     insertion order → stable if you always build in the
                                     same order, unstable if you merge from an unordered source
JS JSON.stringify                 ⚠️ integer keys ascending, string keys in insertion order
                                     → stable for object literals, unstable when converted
                                     from a Map whose source iteration order varies
```

**The culprit is not "language X is broken" but three specific habits**: hand-rolling JSON, accepting a v2 default, and building a collection from an unordered source.

The Python line is the easiest to trip over, because it looks safe:

```python
import json

TOOL_NAMES = ["search", "calculator", "browser", "code_exec"]

# Risky: merged from an unordered source, so insertion order follows upstream
def build_unstable(source):
    tools = {}
    for name, schema in source:        # source order varies -> dict order varies
        tools[name] = schema
    return json.dumps(tools)           # sort_keys defaults to False: insertion order wins

# Safe: alphabetical order, decoupled from how it was built
def build_stable(source):
    tools = {name: schema for name, schema in source}
    return json.dumps(tools, sort_keys=True)
```

`dict` preserving insertion order in Python 3.7+ is a good thing in itself — but it means **the serialised result faithfully reflects the order you built it in.** Upstream order wobbles, the prefix changes. One `sort_keys=True` closes that path.

The Go counterpart:

```go
m := map[string]int{"zebra": 1, "apple": 2, "mango": 3}

b1, _ := json.Marshal(m)
b2, _ := json.Marshal(m)
// string(b1) == string(b2) is always true — the v1 standard library sorts

for k, v := range m {
    // but the order here may differ on every run,
    // so building JSON by hand here makes the prefix unstable
    _ = k; _ = v
}
```

## Culprit 3: the order of the tool list is prefix too

Both vendors write this one down.

Anthropic's framing: modifying a tool definition — name, description or parameters — **invalidates the cache at all three levels, tools, system and messages** — because the prefix hierarchy is `tools → system → messages`, and changing an earlier layer collapses everything after it.

OpenAI names the ordering directly: tool definitions, **tool order**, and the structured-output schema all participate in forming the prompt prefix.

The problem is how many things in a real system can make that order wobble:

| Source | Why it wobbles |
|---|---|
| collecting tools from a dict / map | insertion order varies when merging several sources |
| plugins registering dynamically | registration order follows filesystem walk or import order |
| **the tool list returned by an MCP server** | **the MCP protocol does not specify an ordering for `tools/list`** |
| concurrent registration | several goroutines registering at once; order follows the race |
| conditional enablement by permission or feature flag | contents and order both differ per user and per flag |

The MCP line deserves separate emphasis: a great many agents now pull their tools dynamically from an MCP server, and **there is no ordering guarantee at the protocol level**. Restart the server, or swap the implementation, and the order can change — taking your cache with it.

The fix is trivial, but it has to be explicit: **sort the tools by name after collecting them, before sending.**

```python
tools = sorted(collected_tools, key=lambda t: t["name"])
```

One line, and a whole class of non-determinism is gone.

## Culprit 4: a middle layer silently eating the cache markers

This is the nastiest group, because **your code is correct.**

Several reported LiteLLM issues fall into it:

- **#34797** — forwarding through the SAP provider strips the `cache_control` field, making Anthropic caching simply unavailable
- **#26625** — going through Bedrock Application Inference Profiles on the `/v1/messages` endpoint **silently drops** the `cache_control` directive
- the community has also recorded a release incident that took hit rates from around 90% down to 25–45%

The shared characteristic: **no error, no warning, full functionality.** You wrote the cache breakpoint, the middle layer removed it, the request still succeeds, the answer is still correct, and the bill grows.

The lesson is not "do not use a gateway" — gateways exist for reasons. It is:

> **Any request that passes through any middle layer has to be verified against the final response, not against what your own code says it sent.**

Which leads to the last section.

## How to self-check: two scripts

### Script one: prefix stability

Before spending anything on API calls, verify that the prefix you build is the same every time.

```python
import hashlib, json

def prefix_fingerprint(build_request) -> str:
    """Hash the part of the request that participates in the cache prefix.

    Only what sits before the breakpoint: tools, system, and prior messages.
    """
    req = build_request()
    prefix = {
        "tools": req.get("tools", []),
        "system": req.get("system", ""),
        "messages": req.get("messages", [])[:-1],  # the last one is this turn's input
    }
    blob = json.dumps(prefix, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

# Build it repeatedly; the fingerprint has to be identical every time
prints = {prefix_fingerprint(build_my_request) for _ in range(20)}
if len(prints) > 1:
    raise SystemExit(f"unstable prefix: {len(prints)} distinct fingerprints: {prints}")
print("prefix stable")
```

Note that I used `sort_keys=True` **deliberately** — this script measures whether the prefix is *semantically* identical. If it passes and your real hit rate is still zero, the problem is in your serialisation path (order jitter), not in the content. That distinction halves the search space on its own.

### Script two: the real hit rate

Every vendor names the usage fields differently, and they all share one trap: **`input_tokens` means only the part that was neither read nor written, not the total input.**

```python
def cache_stats(usage: dict, provider: str) -> dict:
    """Reduce each vendor's usage block to the same three numbers."""
    if provider == "anthropic":          # first-party API, or Claude on Vertex / Azure Foundry
        read  = usage.get("cache_read_input_tokens", 0)
        write = usage.get("cache_creation_input_tokens", 0)
        fresh = usage.get("input_tokens", 0)
    elif provider == "bedrock":          # Converse / ConverseStream
        read  = usage.get("cacheReadInputTokens", 0)
        write = usage.get("cacheWriteInputTokens", 0)
        fresh = usage.get("inputTokens", 0)
    elif provider == "openai":           # Responses API; Chat Completions uses prompt_tokens_details
        d     = usage.get("input_tokens_details", {})
        read  = d.get("cached_tokens", 0)
        write = d.get("cache_write_tokens", 0)
        fresh = usage.get("input_tokens", 0)
    elif provider == "gemini":
        read  = usage.get("total_cached_tokens", 0) or usage.get("cachedContentTokenCount", 0)
        write = 0                        # Gemini does not bill cache writes separately
        fresh = usage.get("promptTokenCount", 0) - read
    else:
        raise ValueError(provider)

    total = read + write + fresh
    return {
        "total_input": total,
        "hit_ratio": round(read / total, 3) if total else 0.0,
        "read": read, "write": write, "fresh": fresh,
    }
```

Run that over real traffic and read three signals.

**`read` always 0 and `write` always 0** → nothing is being cached at all. The likeliest cause is a prefix under the minimum token count. **Do not copy the threshold out of the documentation** — I measured Claude Sonnet 4.5 on Bedrock at 1,024 where AWS documents 4,096, off by 4×. Verifying takes two requests; the method and the full data for four models are in [part four](/blog/prompt-cache-4-bedrock-threshold-test/). To measure how many tokens your prompt actually is first, use the [token counter](/tools/token-counter/).

**`write` consistently high while `read` stays low** → the prefix changes on every call. That is the signature of culprits one through three; run script one first to tell a content problem from a serialisation problem.

**Built correctly locally but `read` is 0** → suspect the middle layer. Bypass the proxy once, go direct, and compare the usage for the identical request.

### On the Anthropic first-party API there is an official tool

Anthropic offers **Cache Diagnostics** (beta, request header `cache-diagnosis-2026-04-07`): the API compares two adjacent requests itself and tells you which block the prefix diverged at, with a typed reason — `model_changed`, `system_changed`, `tools_changed`, `messages_changed`, `previous_message_not_found`, and other parameter changes grouped under `unavailable`.

That automates most of the work above. But the documentation states the limitation: **Claude API only, not supported on Bedrock or Google Cloud.**

Which leaves a slightly ironic situation: **the platforms with the highest thresholds and the quietest failures are the ones where the diagnostic tool is unavailable.** On Bedrock and Vertex, script two is your only eyes.

## Defence checklist

In order of return on effort.

**Structure.** Put every static thing first: tool definitions, system prompt, reference documents. Timestamps, session IDs and user input go after the cache breakpoint. Dynamic values you only need for logs go into request metadata.

**Serialisation.** Sort the tool list by name before sending. Use `sort_keys=True` in Python. In Go, do not hand-roll JSON by iterating a map — use `encoding/json` v1, or set `Deterministic` explicitly on v2. In Swift, set `.sortedKeys` explicitly.

**Configuration.** Pin `tool_choice`, the thinking configuration, `effort`, and the structured-output schema. They look like parameters and they end up in the prompt. When evaluating compaction, do the arithmetic: are the tokens saved worth more than the cache thrown away?

**Monitoring.** Push script two's three numbers into your metrics and alert when `read` stays at zero. **That is the only thing that gives a silent failure a voice.**

**Keepalive.** If you genuinely need to ping, use the official warm-up shape — `max_tokens: 0` on Anthropic, output not billed. The interval must be under the TTL, and it has to account for the rule that the TTL is counted from the start of the request with streaming time included. Worth noting: Aider was the first tool to ship cache keepalive publicly, and the interval it uses is five minutes, exactly equal to Anthropic's TTL — with long streamed responses that interval risks overshooting, and past the line every ping becomes a full-price recompute.

## In short

**Invalidation does not error, so it has to be monitored rather than trusted.** Writing `cache_control` in your code does not mean caching happened; a middle layer may already have deleted it.

**Four culprits, by frequency:** volatile content in the prefix, serialisation order jitter, unstable tool ordering, a middle layer silently eating the markers.

**Two scripts cover most investigations:** a prefix fingerprint tells you whether it is stable, the three usage numbers tell you whether it is really hitting. The first one is free.

**Vendor documentation misattributes things too.** Anthropic says Go randomises key order, but the v1 standard library sorts map keys — the real risk is hand-rolled JSON, v2 defaults and unordered sources. A vendor's troubleshooting list is a lead, not a conclusion.

Where to go next: [part one](/blog/prompt-cache-1-how-it-works/) is the mechanism, [part two](/blog/prompt-cache-2-platform-comparison/) compares the bills across platforms, and [part four](/blog/prompt-cache-4-bedrock-threshold-test/) is the measured run on Bedrock that found the wrong threshold this article keeps warning you about. Everything here reflects vendor documentation current in mid-August 2026 — and in this area prices and thresholds change faster than an article goes stale, so verify before you rely on any of it.

---

## Frequently asked

### Does a cache miss raise an error?

No. A prefix that does not match is simply a miss: inference succeeds, the answer is correct, and you are billed at full price with the cache rewritten. A prefix under the minimum length is not even written, and that does not error either. Reading the usage fields in the response is the only way to detect it.

### Is this the same thing as cache penetration?

No. The accurate terms are cache miss and cache invalidation. Cache penetration, in the database sense, means querying a key that does not exist anywhere so the request passes through the cache layer to the database. There is no concept of a non-existent key in prompt caching — either the prefix is there or it is recomputed.

### Does Go's JSON encoder really randomise key order?

The v1 standard library does not: encoding/json sorts map keys, and struct fields follow declaration order. What is actually unstable is hand-rolling JSON by iterating a map (the Go spec explicitly does not guarantee map iteration order) and encoding/json v2, which is non-deterministic unless you set the Deterministic option.

### My code sets cache_control and the hit rate is still zero. What now?

Suspect whatever sits between you and the model. Several LiteLLM issues describe cache_control being stripped or silently dropped by a proxy path, with no error and no warning. Bypass the gateway once, send the identical request direct, and compare the usage fields.


---

## References

- [Prompt caching — Troubleshooting common issues](https://platform.claude.com/docs/en/build-with-claude/prompt-caching) — Anthropic documentation
- [Cache diagnostics](https://platform.claude.com/docs/en/build-with-claude/cache-diagnostics) — Anthropic documentation
- [Prompt caching — Troubleshoot common caching issues](https://platform.openai.com/docs/guides/prompt-caching) — OpenAI documentation
- [encoding/json](https://pkg.go.dev/encoding/json) — Go package documentation
- [The Go Programming Language Specification — For statements with range clause](https://go.dev/ref/spec#For_statements) — Go language specification
- [Caching with Aider](https://aider.chat/docs/usage/caching.html) — Aider documentation