Prompt caching

AI concepts
About 1 min read

An LLM optimization technique that reduces response costs and latency by caching and reusing the repeating prefix of a prompt.

Also known as
Prompt CachingPrompt cacheContext caching

Detailed explanation

Prompt caching is a technique that reduces the cost and latency of reprocessing identical prompt prefixes (such as system instructions, long documents, or examples) across multiple requests by caching and reusing them. It is highly effective for chatbots, agents, and document Q&A systems that repeatedly send a long, shared context. The cached portion is heavily discounted and processed much faster, with caches typically being retained for a short duration. Supported by providers like OpenAI, Anthropic, and Google, it offers significant benefits for services that are cost-sensitive or repeatedly use the same large context.

Why it matters when choosing tools

For services that repeatedly send the same long context (such as system instructions or manuals), prompt caching dramatically reduces costs and improves response speeds. Operational costs depend heavily on whether the tool or API supports caching, how much control you have over what to cache, and the cache retention time and discount rates.

What to look for when choosing a tool

  • Does it support prompt caching, and can you specify what to cache?
  • Are the cost discount rates and latency reductions on cache hits clear?
  • Does the cache retention time align with your usage patterns?
  • Are the retention and access policies for cached sensitive data secure?

Real-world example

For a customer support chatbot that sends a multi-page product manual with every query, caching the manual section significantly reduces processing costs and speeds up response times starting from the second question. This is because only the user's message, which changes each time, needs to be processed anew.

Related terms

Token OptimizationContext WindowLLMLatency