How prompt caching cut our production AI agent input costs by 77%
Inside Deriv’s prompt-caching architecture: deterministic context, longer reusable prefixes, and measured cost reduction at scale.
Take a production AI agent processing one billion input tokens a day, or roughly 30 billion a month.
At GPT-5’s standard input rate, such a workload would cost about $37,500 a month without prompt caching.1
Amy, Deriv’s production AI assistant, operates at a scale where prompt structure has a measurable effect on infrastructure costs. Her actual traffic volume is confidential, but what we measured in production was an 85.8% cache-hit rate, which reduced input-token costs by 77%.
Applied to the same illustrative workload, that measured rate would bring the monthly input bill down to about $8,540.
That is a difference of roughly $29,000 a month.
This article explains the four engineering changes behind that result and how to apply the same structure to your own production agent.
Here is the rule of thumb if you want to skip ahead and price your own agent:
Monthly savings ≈ monthly input tokens (in billions) × cache-hit rate × $1,125
Now, here is the catch: enabling prompt caching does not automatically produce this result.
A system with caching enabled but a poorly organised prompt typically achieves a hit rate of around 20%. On the same workload, it still pays about $30,750 a month. The gap between that system and a well-structured one is $22,210 a month, even though both use the same model and have caching switched on.
We did not close that gap by making Amy less capable or removing the context she needed. We changed the order, stability, and determinism of the context we were already sending.
The expensive part was not only what the prompt contained. It was where we put it.
How prompt caching reduces AI agent costs
Prompt caching allows an LLM provider to reuse computation for an identical prompt prefix. It does not retrieve a previous answer; the model still generates a new response for each request.
Suppose thousands of requests begin with the same system instructions, tool definitions, and operating policies. Processing those tokens from scratch every time is repetitive work.
When the beginning of a new request matches a cached prefix, the provider can reuse the intermediate computation associated with those tokens. At the attention layer, this is commonly described as reusing key-value states (KV states).
Implementation details vary between providers, but the practical rule is simple:
The longer the identical beginning of your prompts, the more computation may be reused.
The pricing difference is significant:
A cached input token costs one-tenth as much as a standard input token. That makes prompt caching valuable. It does not make it automatic magic.
Why prompt caching fails after the first difference
Prompt caches work on prefixes: the sequence of tokens at the beginning of a request.
Imagine two requests:
Request A:
[stable instructions]
[conversation A]
[shared context]
[latest message]Request B:
[stable instructions]
[conversation B]
[shared context]
[latest message]The requests stop matching as soon as their conversation histories diverge. Even though the shared context later in each prompt is identical, it arrives too late to extend the common prefix.
Now reorder the same information:
Request A:
[stable instructions]
[shared context]
[conversation A]
[latest message]Request B:
[stable instructions]
[shared context]
[conversation B]
[latest message]How to structure prompts for higher cache-hit rates
Prompt caches stop being reusable at the first difference between two requests. Placing variable content early shortens the shared prefix, while moving it later allows more input tokens to qualify for the lower cached-input rate.
In the reordered example, the common prefix is longer. The provider has more reusable work before the requests diverge.
This leads to a more precise design principle than “put static content first”:
Order prompt components from least variable to most variable.
To improve prompt-cache hit rate, place stable instructions first, shared context second, conversation-specific content later, and the latest user message last. Repeated blocks must also be rendered deterministically so equivalent requests produce identical token sequences.
“Dynamic” is not a binary property. Some context changes only when a deployment changes. Some changes by organisation or workflow. Conversation history changes by conversation. The latest message changes with every request. A cache-friendly prompt reflects those layers.
Small differences matter. A timestamp, reordered list, inconsistent whitespace, or block rendered in a different sequence can break an otherwise reusable prefix. Two prompts can look equivalent to a human while appearing different to the cache.
Prompt caching therefore depends on three disciplines:
Stable content must come first.
Variable content must come later.
Repeated content must be generated deterministically.
The four steps below show how we applied those disciplines to Amy. Each one is directly transferable to any production agent.
Step 1: Separate static instructions from dynamic context
Amy’s stable instructions live in instruction.md.
We pass those instructions through our Agent Development Kit pipeline as static_instruction. Information that can change between requests is rendered separately through DYNAMIC_CONTEXT_TEMPLATE.
This separation gives us a stable foundation. Amy’s core instructions can remain byte-identical across requests rather than being rebuilt with request-specific data.
It also establishes a useful engineering boundary:
Stable operating instructions belong in the reusable prefix.
Request-dependent context belongs in a separate layer.
Conversation-specific content comes after both.
This was not merely a code-cleanliness improvement. It directly affected how much of each request could be served from cache.
Step 2: Put shared context before conversation history
Our default ADK behaviour placed dynamic instructions near the end of the request, after the conversation history.
That arrangement could support reuse within a single conversation. It was much less effective across different conversations because every conversation has a different history.
Once the histories diverged, any shared content that followed could no longer extend the matching prefix.
We introduced relocate_dynamic_instruction to move the shared dynamic-context block to the beginning of llm_request.contents.
Amy’s effective prompt structure became:
Static instructions
Shared dynamic context
Conversation history
Latest user message
This ordering maximises the common prefix across requests.
The completely stable instructions come first. Context shared by a class of requests comes next. Conversation-specific and request-specific content follows. We did not remove information. We changed its position.
On the illustrative workload, that single ordering decision is where most of the $22,210 monthly gap between a poorly structured prompt and a well-structured one is won or lost.
Step 3: Generate prompt blocks deterministically
Amy can receive prompt blocks describing active and loadable skills. Originally, two requests with the same skills could render those blocks in different orders, depending on how the skills were loaded.
To the model, the meaning was effectively the same. To the cache, the prompts were different. We fixed this by sorting active and loadable skills by skill name before rendering them.
The same skill set now produces the same prompt text in the same order. This makes the skill portion of the prefix byte-identical and, therefore, much more likely to be reused.
This is a common source of invisible cache misses. Sets, maps, database results, tool definitions, and retrieved documents may arrive in a nondeterministic order. If they are rendered directly into a prompt, logically equivalent requests can produce different token sequences.
Deterministic generation is not merely elegant engineering. In a cached LLM system, it has measurable operating value.
Step 4: Monitor prompt-cache hit rate in production
We started measuring Amy’s prompt-cache hit rate in production.
For each request, we compare cached input tokens with total input tokens and track the result over time.
Prompt-cache hit rate = cached input tokens ÷ total input tokens × 100That gave us an immediate feedback loop.
If a prompt or framework change reduces the common prefix, we can see the hit rate fall. If restructuring improves reuse, we can verify the gain against real traffic.
Prompt caching stopped being a feature we had enabled and became a production metric we could optimise.
Amy’s measured cache-hit rate is now 85.8%.
Prompt-caching cost calculation
Here’s how we measured the result:
System: Amy, Deriv’s production AI assistant
Metric: Cached input tokens ÷ total input tokens
Measured cache-hit rate: 85.8%
Cost comparison: Standard versus cached GPT-5 input pricing
Traffic figures: Illustrative because production volume is confidential
Percentages: Based on measured production performance
All figures below use the illustrative workload of 30 billion input tokens per month. The hit rates and percentages are our real, measured results. The dollar figures scale linearly, so you can map them onto any workload.
The calculation is:
Input cost = (non-cached tokens × $1.25 per million)
+ (cached tokens × $0.125 per million)The impact compounds quickly:
At the measured 85.8% hit rate, the savings on this illustrative workload are:
All figures are rounded. Another way to read the same result: for every $10,000 per month you would spend on input tokens without caching, a well-structured prompt saves about $7,700. A poorly structured one saves about $1,800.
Cache-hit rate versus cost reduction
There are three results here that should not be confused with one another:
85.8% is our measured cache-hit rate.
77% is the reduction in input-token cost compared with no caching.
~4× is the value multiple of a well-structured prompt over a poorly structured one: 77% of the savings opportunity captured versus 18%.
The hit rate and cost reduction are not the same because cached tokens are discounted, not free. Amy still pays the cached-input rate for 85.8% of her input and the standard rate for the remaining 14.2%.
The second comparison is especially revealing: A poorly structured prompt still receives some caching benefit. It captures about 18% of the available savings opportunity. A well-structured prompt captures approximately 77% of that opportunity.
Prompt caching was enabled in both cases. Prompt engineering determined how much value was extracted from it.
Does prompt caching reduce output-token costs?
These calculations cover input tokens only.
Output tokens cost the same in all three scenarios because prompt caching does not apply to output generation. The percentage reduction in the total model bill is therefore somewhat lower than the 77% input-cost reduction, but the absolute input saving does not change.
Prompt-caching best practices for production AI agents
Enabling caching is only the starting point
Prompt caching may be a platform feature, but cache performance is an application-design responsibility.
If dynamic content appears near the beginning of every request, the provider cannot reuse the stable content that follows it.
The cache is working correctly. The prompt is simply giving it very little to reuse.
Prompt order is an economic decision
Engineers often treat prompt ordering as a question of model behaviour. At scale, it is also a cost decision.
Moving one block ahead of conversation history allowed more requests to share a longer prefix. That small architectural change materially altered our operating costs.
Determinism has a dollar value
Stable content must be stable in its rendered form, not merely equivalent in meaning.
Sort lists. Canonicalise tool definitions. Keep formatting consistent. Avoid injecting timestamps, request IDs, or other high-cardinality values near the beginning of a prompt.
If the same information can produce different text, it can produce avoidable cache misses.
Cache performance needs regression monitoring
A developer can accidentally reduce cache reuse with an innocent change: moving a block, adding a timestamp, changing whitespace, or rendering tools in a different order.
Without monitoring, the first sign may be a larger invoice.
Cache-hit rate should be tracked like latency, error rate, and token usage—not checked once and forgotten.
Production prompt-caching checklist
Before shipping or restructuring an agent, ask:
What portion of the prompt is identical across nearly all requests?
Can stable instructions and examples be placed first?
Are prompt components ordered from least variable to most variable?
Does shared context appear before conversation-specific history?
Are tools, skills, schemas, and document lists rendered deterministically?
Could timestamps, IDs, or user-specific values be moved later?
Are semantically identical blocks also byte-identical?
Are cached tokens and total input tokens logged for each request?
Is cache-hit rate visible over time and segmented by workflow?
Do deployments include a check for cache-performance regressions?
Are you measuring cost reduction as well as cache-hit rate?
Have you tested the structure against representative production traffic?
Provider-specific thresholds, retention rules, and routing controls can change, so verify them against the relevant platform documentation. The underlying principle remains consistent: maximise the stable prefix and measure what the provider actually reports as cached.
Frequently asked questions about prompt caching
Does prompt caching reuse an earlier response?
No. It reuses computation associated with an identical prompt prefix. The model still generates a new output for each request.
Does enabling prompt caching automatically reduce costs?
Not substantially in every implementation. Cost savings depend on how much of the prompt remains identical across requests and how early the first difference appears.
What breaks a prompt cache?
Changes in timestamps, identifiers, whitespace, ordering, tool definitions, retrieved documents, or any other tokens in the reusable prefix can shorten or eliminate the cache match.
Should conversation history come before shared context?
Usually not when the shared context is common across multiple conversations. Putting conversation-specific history first causes requests to diverge before the shared block can be reused.
Should cache-hit rate be monitored in production?
Yes. Prompt or framework changes can silently lower reuse, so monitor cache-hit rate alongside latency, errors, and token consumption.
What we learned from optimising Amy at Deriv
At Deriv, this is the kind of AI engineering work that matters: not merely demonstrating that an agent can work, but making it reliable, measurable, and economically sustainable in production.
Our 85.8% cache-hit rate came from separating static and dynamic instructions, ordering prompt components by variability, generating repeated content deterministically, and monitoring the result against real traffic.
Prompt caching is a platform capability. Cache efficiency is an engineering outcome.
Your agent’s prompt is no longer just a string. It is part of your production architecture, and its structure will eventually appear on your invoice.
Interested in solving production AI problems at this scale? Explore AI and engineering careers at Deriv.
Osama Ghazal is a Senior AI Engineer at Deriv.
Follow our official LinkedIn page for company updates and upcoming events.
All pricing used in this article was verified on 15 July 2026. Provider pricing, cache thresholds, and retention policies may change.
 |
|