Prompt Caching (Part 3): It Is On and Not Hitting — Now What

Verdict first.

Prompt cache invalidation almost never raises an error. Inference succeeds, the answer is correct, the logs are clean, and the bill goes up. That is what makes it hard to find, and it is the reason this is one of the few things you have to monitor deliberately rather than trust.

Ordered by how often I have actually seen them, the culprits fall into four groups:

1. Volatile content inside the prefix — timestamps, session IDs, context compaction. The most common, and the easiest to fix.
2. Serialisation order jitter — the same data structure producing different bytes on two passes.
3. Unstable tool ordering — tools is part of the prefix, and its order is often decided by the filesystem or the thread scheduler.
4. A middle layer silently eating the cache markers — you set cache_control, a proxy removed it. The nastiest of the four.

If you already have a service you suspect is not hitting, skip to the self-check scripts at the end; five minutes gets you an answer.

Part one covered the mechanism and part two the platform differences. This part is the debugging.

Calibrate the terminology first#

If your instinct comes from the Redis-and-MySQL world, it is worth separating two things that sound alike:

Prompt caching has no notion of a non-existent key — the prefix is either there and hit, or absent and recomputed. Calling it invalidation, or just a miss, keeps the investigation pointed in the right direction.

Culprit 1: volatile content inside the prefix#

This accounts for most of what I have seen, and the reason is the plainest: attention is causal, so every KV after an edit is void. Anything volatile sitting near the front zeroes the whole cache.

All three vendors name specific cases in their documentation.

A dynamic timestamp at the top of the prompt. The classic. "Current time: 2026-08-17 10:25:07" at the head of a system prompt changes every second, and the hit rate is identically zero.

A session ID or user ID in the system prompt. Different for every conversation, so every conversation writes the cache afresh and never reads it. OpenAI’s advice here is the practical one: a dynamic value you only need for logging or debugging belongs in request metadata, not in the prompt.

Context compaction resets the prefix. This one gets missed because compaction is itself a good optimisation. OpenAI’s documentation is clear about it: truncation, summarisation and compaction reduce prompt size but also reset the reusable prefix. Two optimisations fighting each other — the tokens you save by compacting may be fewer than the tokens you now pay full price for. That trade-off has to be computed on your own data; there is no general answer.

Rewriting or reordering history. Multi-turn conversations should append, not edit. Delete an early message, or reorder the history, and the prefix has changed.

Beyond those, there is a whole class of “parameters that turn out to be prefix”, all of them documented:

• image order, or a change to the detail parameter
• a change to the structured-output schema
• a change to the thinking configuration or budget_tokens
• a change to output_config.effort
• a change to tool_choice
• toggling web search or citations, which rewrite the system prompt
• switching between fast mode and standard speed

What those last few have in common: they look like request parameters and are actually rendered into the prompt. An A/B experiment that varies the thinking budget will miss the cache for its entire duration.

Culprit 2: serialisation order jitter — and one vendor misattribution#

Anthropic’s troubleshooting section carries a warning to the effect that you should make sure the key order inside tool_use content blocks is stable, because some languages — Swift and Go are named — randomise key order during JSON conversion and break the cache.

The direction is right. Naming languages is not. I went and checked Go’s own documentation.

The encoding/json page on pkg.go.dev is explicit: map keys are sorted before being used as JSON object keys. The v1/v2 differences section goes further — in v1, Go maps serialise in a deterministic order, while v2 is non-deterministic and controlled by the jsonv2.Deterministic option.

So the v1 standard library is precisely the safe case. The real risk distribution looks like this:

json.Marshal(someMap)             ✅ keys already sorted, output stable
json.Marshal(someStruct)          ✅ declaration order, output stable
hand-rolled for k, v := range m   ❌ the Go spec guarantees no map iteration order
encoding/json v2                  ❌ non-deterministic by default, needs Deterministic
Swift JSONEncoder                 ❌ unsorted by default (.sortedKeys is opt-in)
Python json.dumps                 ⚠️ sort_keys defaults to False, but 3.7+ dicts keep
                                     insertion order → stable if you always build in the
                                     same order, unstable if you merge from an unordered source
JS JSON.stringify                 ⚠️ integer keys ascending, string keys in insertion order
                                     → stable for object literals, unstable when converted
                                     from a Map whose source iteration order varies

The culprit is not “language X is broken” but three specific habits: hand-rolling JSON, accepting a v2 default, and building a collection from an unordered source.

The Python line is the easiest to trip over, because it looks safe:

import json

TOOL_NAMES = ["search", "calculator", "browser", "code_exec"]

# Risky: merged from an unordered source, so insertion order follows upstream
def build_unstable(source):
    tools = {}
    for name, schema in source:        # source order varies -> dict order varies
        tools[name] = schema
    return json.dumps(tools)           # sort_keys defaults to False: insertion order wins

# Safe: alphabetical order, decoupled from how it was built
def build_stable(source):
    tools = {name: schema for name, schema in source}
    return json.dumps(tools, sort_keys=True)

dict preserving insertion order in Python 3.7+ is a good thing in itself — but it means the serialised result faithfully reflects the order you built it in. Upstream order wobbles, the prefix changes. One sort_keys=True closes that path.

The Go counterpart:

m := map[string]int{"zebra": 1, "apple": 2, "mango": 3}

b1, _ := json.Marshal(m)
b2, _ := json.Marshal(m)
// string(b1) == string(b2) is always true — the v1 standard library sorts

for k, v := range m {
    // but the order here may differ on every run,
    // so building JSON by hand here makes the prefix unstable
    _ = k; _ = v
}

Culprit 3: the order of the tool list is prefix too#

Both vendors write this one down.

Anthropic’s framing: modifying a tool definition — name, description or parameters — invalidates the cache at all three levels, tools, system and messages — because the prefix hierarchy is tools → system → messages, and changing an earlier layer collapses everything after it.

OpenAI names the ordering directly: tool definitions, tool order, and the structured-output schema all participate in forming the prompt prefix.

The problem is how many things in a real system can make that order wobble:

The MCP line deserves separate emphasis: a great many agents now pull their tools dynamically from an MCP server, and there is no ordering guarantee at the protocol level. Restart the server, or swap the implementation, and the order can change — taking your cache with it.

The fix is trivial, but it has to be explicit: sort the tools by name after collecting them, before sending.

tools = sorted(collected_tools, key=lambda t: t["name"])

One line, and a whole class of non-determinism is gone.

Culprit 4: a middle layer silently eating the cache markers#

This is the nastiest group, because your code is correct.

Several reported LiteLLM issues fall into it:

• #34797 — forwarding through the SAP provider strips the cache_control field, making Anthropic caching simply unavailable
• #26625 — going through Bedrock Application Inference Profiles on the /v1/messages endpoint silently drops the cache_control directive
• the community has also recorded a release incident that took hit rates from around 90% down to 25–45%

The shared characteristic: no error, no warning, full functionality. You wrote the cache breakpoint, the middle layer removed it, the request still succeeds, the answer is still correct, and the bill grows.

The lesson is not “do not use a gateway” — gateways exist for reasons. It is:

Any request that passes through any middle layer has to be verified against the final response, not against what your own code says it sent.

Which leads to the last section.

How to self-check: two scripts#

Script one: prefix stability#

Before spending anything on API calls, verify that the prefix you build is the same every time.

import hashlib, json

def prefix_fingerprint(build_request) -> str:
    """Hash the part of the request that participates in the cache prefix.

    Only what sits before the breakpoint: tools, system, and prior messages.
    """
    req = build_request()
    prefix = {
        "tools": req.get("tools", []),
        "system": req.get("system", ""),
        "messages": req.get("messages", [])[:-1],  # the last one is this turn's input
    }
    blob = json.dumps(prefix, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(blob.encode()).hexdigest()[:16]

# Build it repeatedly; the fingerprint has to be identical every time
prints = {prefix_fingerprint(build_my_request) for _ in range(20)}
if len(prints) > 1:
    raise SystemExit(f"unstable prefix: {len(prints)} distinct fingerprints: {prints}")
print("prefix stable")

Note that I used sort_keys=True deliberately — this script measures whether the prefix is semantically identical. If it passes and your real hit rate is still zero, the problem is in your serialisation path (order jitter), not in the content. That distinction halves the search space on its own.

Script two: the real hit rate#

Every vendor names the usage fields differently, and they all share one trap: input_tokens means only the part that was neither read nor written, not the total input.

def cache_stats(usage: dict, provider: str) -> dict:
    """Reduce each vendor's usage block to the same three numbers."""
    if provider == "anthropic":          # first-party API, or Claude on Vertex / Azure Foundry
        read  = usage.get("cache_read_input_tokens", 0)
        write = usage.get("cache_creation_input_tokens", 0)
        fresh = usage.get("input_tokens", 0)
    elif provider == "bedrock":          # Converse / ConverseStream
        read  = usage.get("cacheReadInputTokens", 0)
        write = usage.get("cacheWriteInputTokens", 0)
        fresh = usage.get("inputTokens", 0)
    elif provider == "openai":           # Responses API; Chat Completions uses prompt_tokens_details
        d     = usage.get("input_tokens_details", {})
        read  = d.get("cached_tokens", 0)
        write = d.get("cache_write_tokens", 0)
        fresh = usage.get("input_tokens", 0)
    elif provider == "gemini":
        read  = usage.get("total_cached_tokens", 0) or usage.get("cachedContentTokenCount", 0)
        write = 0                        # Gemini does not bill cache writes separately
        fresh = usage.get("promptTokenCount", 0) - read
    else:
        raise ValueError(provider)

    total = read + write + fresh
    return {
        "total_input": total,
        "hit_ratio": round(read / total, 3) if total else 0.0,
        "read": read, "write": write, "fresh": fresh,
    }

Run that over real traffic and read three signals.

read always 0 and write always 0 → nothing is being cached at all. The likeliest cause is a prefix under the minimum token count. Do not copy the threshold out of the documentation — I measured Claude Sonnet 4.5 on Bedrock at 1,024 where AWS documents 4,096, off by 4×. Verifying takes two requests; the method and the full data for four models are in part four. To measure how many tokens your prompt actually is first, use the token counter.

write consistently high while read stays low → the prefix changes on every call. That is the signature of culprits one through three; run script one first to tell a content problem from a serialisation problem.

Built correctly locally but read is 0 → suspect the middle layer. Bypass the proxy once, go direct, and compare the usage for the identical request.

On the Anthropic first-party API there is an official tool#

Anthropic offers Cache Diagnostics (beta, request header cache-diagnosis-2026-04-07): the API compares two adjacent requests itself and tells you which block the prefix diverged at, with a typed reason — model_changed, system_changed, tools_changed, messages_changed, previous_message_not_found, and other parameter changes grouped under unavailable.

That automates most of the work above. But the documentation states the limitation: Claude API only, not supported on Bedrock or Google Cloud.

Which leaves a slightly ironic situation: the platforms with the highest thresholds and the quietest failures are the ones where the diagnostic tool is unavailable. On Bedrock and Vertex, script two is your only eyes.

Defence checklist#

In order of return on effort.

Structure. Put every static thing first: tool definitions, system prompt, reference documents. Timestamps, session IDs and user input go after the cache breakpoint. Dynamic values you only need for logs go into request metadata.

Serialisation. Sort the tool list by name before sending. Use sort_keys=True in Python. In Go, do not hand-roll JSON by iterating a map — use encoding/json v1, or set Deterministic explicitly on v2. In Swift, set .sortedKeys explicitly.

Configuration. Pin tool_choice, the thinking configuration, effort, and the structured-output schema. They look like parameters and they end up in the prompt. When evaluating compaction, do the arithmetic: are the tokens saved worth more than the cache thrown away?

Monitoring. Push script two’s three numbers into your metrics and alert when read stays at zero. That is the only thing that gives a silent failure a voice.

Keepalive. If you genuinely need to ping, use the official warm-up shape — max_tokens: 0 on Anthropic, output not billed. The interval must be under the TTL, and it has to account for the rule that the TTL is counted from the start of the request with streaming time included. Worth noting: Aider was the first tool to ship cache keepalive publicly, and the interval it uses is five minutes, exactly equal to Anthropic’s TTL — with long streamed responses that interval risks overshooting, and past the line every ping becomes a full-price recompute.

In short#

Invalidation does not error, so it has to be monitored rather than trusted. Writing cache_control in your code does not mean caching happened; a middle layer may already have deleted it.

Four culprits, by frequency: volatile content in the prefix, serialisation order jitter, unstable tool ordering, a middle layer silently eating the markers.

Two scripts cover most investigations: a prefix fingerprint tells you whether it is stable, the three usage numbers tell you whether it is really hitting. The first one is free.

Vendor documentation misattributes things too. Anthropic says Go randomises key order, but the v1 standard library sorts map keys — the real risk is hand-rolled JSON, v2 defaults and unordered sources. A vendor’s troubleshooting list is a lead, not a conclusion.

Where to go next: part one is the mechanism, part two compares the bills across platforms, and part four is the measured run on Bedrock that found the wrong threshold this article keeps warning you about. Everything here reflects vendor documentation current in mid-August 2026 — and in this area prices and thresholds change faster than an article goes stale, so verify before you rely on any of it.