The hit rate dropped, and the total alone couldn’t say why. Looked at in 128-token blocks, the losses came from one section placed in the wrong spot, and from two providers with completely different caching rules.
Table of Contents
Every turn, Autarkia sends the model a long request: a persona definition several thousand characters long, then conversation history, then content specific to this turn. Providers charge far less for input tokens that hit the prefix cache, so the more of the prefix gets reused, the smaller the bill.
I had already been burned here once: the cache anchor was tied to a quantity that upstream truncated into a constant, so the quantization never did anything (the story is in “Fixed Three Times, Still Broken”). Since then I had been watching the total hit rate, but the total can’t show which section is changing.
01The cache works in blocks
First you need the cache’s granularity. I worked it out from 2,840 log lines carrying cached-token counts: 99.3% were multiples of 128, only 49.6% multiples of 256, and the smallest hit was exactly 128. The cache works in 128-token blocks, compared from the first byte; whichever block changes, everything from that block onward is recomputed.
So I added a layer of instrumentation that observes without touching the request: after each prefix is assembled, cut it into 128-token blocks and record which blocks match the previous turn, which changed, and which section the break falls in. Sessions are stored as hashes, never raw user IDs, and the volatile section is recorded only as a fingerprint. A companion script compares adjacent turns per session and counts how often breaks happen and how many blocks each one wastes.
02One section in the wrong place
The instrumentation showed the problem right away. The function that assembles the prefix says in its header comment that the order is “system → history → dynamic tail”, but the code inserted the retrieved-material section between the system definition and the history. That section changes almost every turn, and whenever it does, the entire history after it is recomputed. The longer the history, the more each break wastes.
The fix looks like moving one section, but this round I only added observation and left the order alone. Whether the material sits before or after the history changes where the model encounters it, and with that whether and how it uses the material. That’s a behaviour change; it needs an A/B on reply quality before anything moves.
Estimating token counts had its own trap. The context budget uses a very conservative conversion so it never overflows, which overestimates Chinese token counts by more than double. Cut blocks with that and the block count doubles too, putting every break in the wrong place. The instrumentation uses its own ratio (about 1.5 Chinese characters per token) and writes the ratio into the log for later calibration.
03Two providers, two sets of rules
The second finding had nothing to do with the code. On DeepSeek, what I observed is that the cache is isolated per API key: when several roles share a default key, their prefixes evict one another. With a dedicated key per role, each role’s prefix stays byte-for-byte fixed, and nightly memory extraction on new segments hit 95.4% to 97.9% across four roles.
Put the same prefix layout on MiniMax’s subscription plan and it only hits when the entire request is identical: 10.9% over one batch of 31 segments, around 2% on new segments. The plan doesn’t offer per-role keys either, so there’s nothing to tune at that layer.
So when I compare providers now, I measure the cache hit rate with my own prompts separately; comparing unit prices alone gives the wrong answer.
Nightly extraction hits that high also because it deliberately reuses the main chat’s persona prefix: the offline build goes through the same templates and the same assembly order as the main chat, byte for byte, with the extraction instructions appended after. On a small sample, one two-segment extraction had 6,528 of its 6,997 input tokens served from cache.