Prompt Caching Explained: Cut LLM Costs up to 90% (2026)

Prompt Caching Explained: Cut LLM Costs up to 90%

The snippet answer first: prompt caching tells the model provider to reuse a repeated prefix of your prompt - your system prompt, tool definitions, RAG documents, or few-shot examples - so those input tokens are billed at a steep discount instead of full price on every call. On a cache hit, the cached blocks can cost up to 90% less. It is the single biggest cost lever for any workload that resends the same context over and over.

If you run a chatbot, a RAG pipeline, or an agent loop, you are almost certainly paying full price for tokens you have already sent hundreds of times. This is the practical guide to fixing that.

What is prompt caching?

Prompt caching is a provider-native feature that stores the processed form of a repeated prompt prefix so it does not have to be re-processed and re-billed on the next request. Every LLM call has two parts: the stable stuff you send every time (instructions, tools, retrieved context) and the variable stuff that changes per request (the user’s actual question). Prompt caching targets the first part.

Here is the mechanism. When a new request shares an identical prefix with a recent one, the provider serves that prefix from cache and charges you a fraction of the normal input-token rate for it. You still send the whole prompt; you just are not billed full price for the parts the provider has already seen.

Two things to keep straight:

• The discount applies only to the cached prefix, not to output tokens and not to the variable tail of the prompt. If 90% of your request is a stable prefix, your savings are large. If 10% is, they are modest.
• The prefix must be byte-stable. Caching matches on an exact prefix. Change one character near the front - a timestamp, a reordered document, a tweaked instruction - and the cache misses, and you pay full price again.

How much does prompt caching save?

On a cache hit, the cached portion of your input can cost up to 90% less than uncached input tokens. The headline number is real, but the honest version has a denominator: it applies to cached input tokens only.

Work an example. Say each chatbot turn sends a 4,000-token system-plus-tools prefix and a 200-token user message. Without caching, you pay full input price on all 4,200 tokens every turn. With caching, after the first call the 4,000-token prefix is discounted heavily and you pay full price only on the 200-token tail. Across a busy day of thousands of turns, that is the difference between a comfortable bill and a scary one. We break down the full cost stack - caching, routing, and compression together - in our guide to cutting LLM costs for production chatbots and agents.

The workloads that benefit most:

• RAG pipelines that stuff the same retrieved documents into context across many questions.
• AI agents that resend a large system prompt and tool schema on every single turn of a loop.
• Few-shot prompts with long, fixed example sets.
• Long-document Q and A where users ask many questions about one large document.

How do OpenAI, Anthropic, and Google differ?

All three major providers offer prompt caching, but the developer experience differs. The core split is automatic versus explicit control.

The practical read: with OpenAI, structure your prompt well and caching happens for free. With Anthropic, you get finer control - you decide which blocks are cache breakpoints, which is powerful when only part of your prefix is stable. With Google Gemini, context caching is the lever for large reused context. In all three, the same design discipline wins: a long, stable, repeated prefix.

Always confirm the current TTL and exact pricing in each provider’s docs before you commit numbers to a budget. Cache entries expire, and a cache that has aged out is a cache miss.

Prompt caching vs semantic caching?

This is the confusion that costs teams money, so be precise: prompt caching and semantic caching are different tools that stack.

• Prompt caching discounts the repeated input prefix but still calls the model. You get a cheaper call, not zero call.
• Semantic caching skips the model call entirely. When a new query is semantically similar to one you have answered before, you return the stored answer with no inference at all.

They are complementary. Prompt caching makes the calls you make cheaper; semantic caching removes the calls you should not make. A chatbot answering the same top-20 questions all day should use semantic caching to short-circuit those, and prompt caching to cheapen everything that does reach the model. For the tooling side of that second layer, see our comparison of semantic caching for LLM chatbots with GPTCache, Portkey, and Redis.

Add a third lever, model routing, and you have the full cost stack: route easy turns to a cheaper model, cache the prefixes, and skip repeats entirely. Our breakdown of RouteLLM vs OpenRouter vs Not Diamond covers where routing fits.

How do you structure prompts to maximize cache hits?

One rule does most of the work: put stable content first and variable content last. The cacheable prefix runs from the start of the prompt up to the first thing that changes, so anything variable near the front shrinks or kills your cache.

Order your prompt like this:

[1] System prompt        <- stable, cache-friendly
[2] Tool / function defs  <- stable
[3] Retrieved RAG docs    <- stable per session
[4] Few-shot examples     <- stable
--- cache breakpoint ---
[5] Conversation history  <- grows, but append-only
[6] Current user message  <- variable, always last

Everything from [1] through [4] is a byte-stable prefix that caches beautifully. The user’s actual question sits at the very end, where it changes without disturbing the prefix.

Practical rules that follow from this:

• Keep prefixes byte-stable. No timestamps, request IDs, or shuffled document order in the cached region. Sort retrieved documents deterministically so the same set produces the same bytes.
• Append, never rewrite. Conversation history should grow by appending turns, so the earlier, cached portion stays identical.
• Watch the TTL. Caches expire. Bursty traffic keeps entries warm; long gaps let them lapse into cache misses.
• Cache the tool schema. Agents resend large tool definitions every turn - a prime, high-value cache target. This pairs naturally with the stateless MCP core in the 2026-07-28 spec, where a stable tool and context layer is exactly what you want cached across requests.

There is a real caveat for agent builders. Recent research (“Don’t Break the Cache”) shows that long-horizon agentic tasks can quietly lose cache efficiency when tool outputs keep changing the prefix mid-run. If every tool result is injected near the top of the context, each call busts the cache. The fix is the same principle applied harder: keep system and tool context stable at the front, and push volatile tool outputs toward the end where they belong.

What about content that is not cacheable?

Not every workload has a fat, stable prefix. When your context genuinely changes per request, reach for prompt compression instead. Techniques like LLMLingua shrink the input token count on long-document tasks - often by 30 to 40% - with no measurable quality loss, by pruning tokens the model does not need to produce the same answer.

Compression and caching are complementary, not competing. Cache what is stable; compress what is not. A RAG pipeline might cache the system prompt and tool layer while compressing the freshly retrieved documents that differ on every query. The two together attack both halves of the input bill.

What this means for UAE and GCC teams

For teams shipping AI products in the UAE and wider GCC, prompt caching is one of the fastest paths to a defensible unit economic. Local budgets scrutinize per-conversation cost, and caching turns a linear token bill into something closer to a fixed prefix plus a small variable tail - the difference between a pilot that pencils out and one that stalls at the finance review.

It also reads well in a governance conversation. Caching is a transparent, auditable optimization: you can show exactly which parts of a prompt are cached, that no output tokens are discounted, and that customer-specific variable content sits outside the shared prefix. Under the UAE Personal Data Protection Law (PDPL), keeping personal data in the variable tail rather than baked into a shared cached prefix is also cleaner data hygiene. As always, confirm each provider’s cache retention behavior against your data-residency and retention requirements.

The bottom line

Prompt caching is the highest-leverage, lowest-effort cost lever for any LLM workload that resends the same context - chatbots, RAG, and agents especially. Get the up-to-90% discount by doing three things: put stable content first and variable content last, keep the prefix byte-stable, and layer semantic caching and prompt compression on top so you cheapen the calls you make, skip the ones you do not, and shrink the context you cannot cache.

NomadX is an AI agents consultancy in Dubai that builds cost-efficient production agents for UAE and GCC enterprises. If you want a caching, routing, and compression strategy grounded in your actual traffic and token bill rather than a generic checklist - book a free 30-minute consultation.