Claude Prompt Caching

curl https://aihubmix.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: AIHUBMIX_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "stream": true,
    "model": "claude-opus-4-20250514",
    "max_tokens": 20000,
    "system": [
      {
        "type": "text",
        "text": "You are an AI assistant tasked with analyzing literary works. Your goal is to provide insightful commentary on themes, characters, and writing style."
      },
      {
        "type": "text",
        "text": "Pride and Prejudice by Jane Austen... [Place complete text content here]",
        "cache_control": {"type": "ephemeral"}
      }
    ],
    "thinking": {
      "type": "enabled",
      "budget_tokens": 16000
    },
    "messages": [
      {
        "role": "user",
        "content": "Analyze the major themes in Pride and Prejudice."
      }
    ]
  }'

import os
import anthropic

client = anthropic.Anthropic(
    api_key="sk-***", # Replace with the key you generated in AiHubMix
    base_url="https://aihubmix.com"
)

# Streaming response with caching
with client.messages.stream(
    model="claude-opus-4-20250514",
    max_tokens=20000,
    system=[
        {
            "type": "text",
            "text": "You are an AI assistant tasked with analyzing literary works. Your goal is to provide insightful commentary on themes, characters, and writing style.\n"
        },
        {
            "type": "text",
            "text": "<the entire contents of 'Pride and Prejudice'>",
            "cache_control": {"type": "ephemeral"}
        }
    ],
    thinking={
        "type": "enabled",
        "budget_tokens": 16000
    },
    messages=[
        {"role": "user", "content": "Analyze the major themes in 'Pride and Prejudice'."}
    ]
) as stream:
    for text in stream.text_stream:
        print(text, end="", flush=True)

# Non-streaming response
message = client.messages.create(
    model="claude-opus-4-20250514",
    max_tokens=20000,
    system=[
        {
            "type": "text",
            "text": "You are an AI assistant tasked with analyzing literary works."
        },
        {
            "type": "text",
            "text": "<the entire contents of 'Pride and Prejudice'>",
            "cache_control": {"type": "ephemeral"}
        }
    ],
    messages=[
        {"role": "user", "content": "Analyze the major themes in 'Pride and Prejudice'."}
    ]
)
print(message.content)

import requests

url = "https://aihubmix.com/v1/messages"
headers = {
    "content-type": "application/json",
    "x-api-key": "sk-***", # Replace with the key you generated in AiHubMix
    "anthropic-version": "2023-06-01"
}
data = {
    "stream": True,
    "model": "claude-opus-4-20250514",
    "max_tokens": 20000,
    "system": [
        {
            "type": "text",
            "text": "You are an AI assistant tasked with analyzing literary works. Your goal is to provide insightful commentary on themes, characters, and writing style.\n"
        },
        {
            "type": "text",
            "text": "<the entire contents of 'Pride and Prejudice'>",
            "cache_control": {"type": "ephemeral"}
        }
    ],
    "thinking": {
        "type": "enabled",
        "budget_tokens": 16000
    },
    "messages": [{"role": "user", "content": "Analyze the major themes in 'Pride and Prejudice'."}]
}

response = requests.post(url, headers=headers, json=data, stream=True)

# Check response status
if response.status_code == 200:
    # Process the streaming response
    for line in response.iter_lines():
        if line:
            print(line.decode('utf-8'))
else:
    print(f"Error: {response.status_code}, {response.text}")

{"cache_creation_input_tokens":188086,"cache_read_input_tokens":0,"input_tokens":21,"output_tokens":393}
{"cache_creation_input_tokens":0,"cache_read_input_tokens":188086,"input_tokens":21,"output_tokens":393}

​

How prompt caching works

1. The system checks if a prompt prefix, up to a specified cache breakpoint, is already cached from a recent query.
2. If found, it uses the cached version, reducing processing time and costs.
3. Otherwise, it processes the full prompt and caches the prefix once the response begins. This is especially useful for:

• Prompts with many examples
• Large amounts of context or background information
• Repetitive tasks with consistent instructions
• Long multi-turn conversations

​

Common mistake: “writing the cache but never reading it”

​

❌ Wrong: putting the per-turn question before the breakpoint

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "请总结这份资料的核心观点。" },                          // ← changes every turn, yet placed before the breakpoint
        { "type": "text", "text": "<大文档>", "cache_control": { "type": "ephemeral" } }   // breakpoint
      ]
    }
  ]
}

​

✅ Correct: large document first + breakpoint + question last

{
  "messages": [
    {
      "role": "user",
      "content": [
        { "type": "text", "text": "<固定不变的大文档/参考资料，≥4096 token>", "cache_control": { "type": "ephemeral" } },  // breakpoint; prefix stays constant
        { "type": "text", "text": "请总结这份资料的核心观点。" }                                                          // ← the per-turn question, placed after the breakpoint
      ]
    }
  ]
}

​

Measured comparison (claude-opus-4-6, a few seconds between the two calls)

1. Put the fixed, large block (reference document, long context) at the very front of the messages user message, with cache_control at its end, and do not change a single character of it;
2. Put the per-turn question/instruction after the breakpoint (after the large document within the same user message, or in subsequent messages); in multi-turn conversations, only append — never go back and edit earlier messages;
3. When thinking is enabled, thinking blocks in earlier assistant turns must be passed back verbatim, otherwise the prefix breaks the same way (see “What cannot be cached” below);
4. If a block is smaller than the minimum cache threshold (which varies by model, from 512 to 4,096 tokens — see “Cache limitations” below), it won’t be written to the cache even if marked with cache_control — this is expected behavior, see “Cache limitations” below.

​

Pricing

• 5-minute cache write tokens are 1.25 times the base input tokens price
• 1-hour cache write tokens are 2 times the base input tokens price
• Cache read tokens are 0.1 times the base input tokens price
• Regular input and output tokens are priced at platform standard rates

​

How to implement prompt caching

​

Supported models

​

Automatic caching (top-level cache_control)

{
  "model": "claude-sonnet-5",
  "max_tokens": 1024,
  "cache_control": {"type": "ephemeral"},
  "system": "You are an AI assistant tasked with analyzing literary works.",
  "messages": [
    {"role": "user", "content": "Analyze the major themes in Pride and Prejudice."}
  ]
}

​

Structuring your prompt

​

Cache limitations

• “ephemeral”: Default 5-minute lifetime
• 1-hour cache: Set "ttl": "1h" in cache_control, for scenarios requiring longer cache duration

​

1-hour cache duration

curl https://aihubmix.com/v1/messages \
  -H "content-type: application/json" \
  -H "x-api-key: AIHUBMIX_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-opus-4-20250514",
    "system": [
      {
        "type": "text",
        "text": "Long-term instructions...",
        "cache_control": {
          "type": "ephemeral",
          "ttl": "1h"
        }
      }
    ],
    "messages": [...]
  }'

{
  "cache_control": {
    "type": "ephemeral",
    "ttl": "5m" | "1h"
  }
}

​

When to use 1-hour cache

• Batch processing: Processing large volumes of requests with common prefixes
• Long-running sessions: Conversations requiring context maintenance over extended periods
• Large document analysis: Multiple different types of analysis on the same document
• Codebase Q&A: Multiple queries on the same codebase over extended periods

​

Mixing different TTLs

{
  "system": [
    {
      "type": "text", 
      "text": "Long-term instructions...",
      "cache_control": {
        "type": "ephemeral",
        "ttl": "1h"
      }
    },
    {
      "type": "text",
      "text": "Short-term context...", 
      "cache_control": {
        "type": "ephemeral",
        "ttl": "5m"
      }
    }
  ]
}

​

What can be cached

• Tools: Tool definitions in the tools array
• System messages: Content blocks in the system array
• Messages: Content blocks in the messages.content array, for both user and assistant turns
• Images & Documents: Content blocks in the messages.content array, in user turns
• Tool use and tool results: Content blocks in the messages.content array, in both user and assistant turns

​

What cannot be cached

• Thinking blocks cannot be cached directly with cache_control. However, thinking blocks CAN be cached alongside other content when they appear in previous assistant turns. When cached this way, they DO count as input tokens when read from cache.
• Sub-content blocks (like citations) themselves cannot be cached directly. Instead, cache the top-level block.
• Empty text blocks cannot be cached.

​

Tracking cache performance

• cache_creation_input_tokens: Number of tokens written to the cache when creating a new entry.
• cache_read_input_tokens: Number of tokens retrieved from the cache for this request.
• input_tokens: Number of input tokens which were not read from or used to create a cache.

​

Best practices for effective caching

• Cache stable, reusable content like system instructions, background information, large contexts, or frequent tool definitions.
• Place cached content at the prompt’s beginning for best performance.
• Use cache breakpoints strategically to separate different cacheable prefix sections.
• Regularly analyze cache hit rates and adjust your strategy as needed.
• For long-term content, consider using 1-hour cache for better cost efficiency.

​

Optimizing for different use cases

• Conversational agents: Reduce cost and latency for extended conversations, especially those with long instructions or uploaded documents.
• Coding assistants: Improve autocomplete and codebase Q&A by keeping relevant sections or a summarized version of the codebase in the prompt.
• Large document processing: Incorporate complete long-form material including images in your prompt without increasing response latency.
• Detailed instruction sets: Share extensive lists of instructions, procedures, and examples to fine-tune Claude’s responses. Developers often include an example or two in the prompt, but with prompt caching you can get even better performance by including 20+ diverse examples of high quality answers.
• Agentic tool use: Enhance performance for scenarios involving multiple tool calls and iterative code changes, where each step typically requires a new API call.
• Talk to books, papers, documentation, podcast transcripts, and other longform content: Bring any knowledge base alive by embedding the entire document(s) into the prompt, and letting users ask it questions.

​

Troubleshooting common issues

• Ensure cached sections are identical and marked with cache_control in the same locations across calls
• Check that calls are made within the cache lifetime (5 minutes or 1 hour)
• Verify that tool_choice and image usage remain consistent between calls
• Validate that you are caching at least the minimum number of tokens
• While the system will attempt to use previously cached content at positions prior to a cache breakpoint, you may use an additional cache_control parameter to guarantee cache lookup on previous portions of the prompt, which may be useful for queries with very long lists of content blocks

​

Cache storage and sharing

• Organization Isolation: Caches are isolated between organizations. Different organizations never share caches, even if they use identical prompts.
• Exact Matching: Cache hits require 100% identical prompt segments, including all text and images up to and including the block marked with cache control. The same block must be marked with cache_control during cache reads and creation.
• Output Token Generation: Prompt caching has no effect on output token generation. The response you receive will be identical to what you would get if prompt caching was not used.

​

Enabling Claude caching in clients / platforms

​

Dify (via the Aihubmix plugin)

1. Wrap the prompt you want to cache (the fixed system prompt / long context) in <cache>…</cache>, and the plugin will automatically convert it into a cache_control breakpoint at that point;
2. In the model parameters, set the “auto-cache threshold for large messages” to a positive integer: the cache is only actually written once the content reaches that token threshold (still subject to the minimum cache requirement in “Cache limitations” below, which is 4096 tokens for Opus 4.5/4.6 and Haiku 4.5); setting it to 0 or leaving it blank disables it.

​

Cherry Studio

1. Click the gear to the right of the Aihubmix provider name to open “API Settings”:

2. Configure the following three items, and the client will automatically inject cache_control for Claude accordingly:

• Cache Token Threshold: a cache breakpoint is only injected once the content exceeds this number of tokens (set a positive number to enable, 0 or blank to disable);
• Cache System Message: when enabled, a cache breakpoint is placed on the system message (good for caching a fixed long system prompt);
• Cache Last N Messages: places a cache breakpoint on the most recent N messages (good for rolling caching in multi-turn conversations).

​

FAQ

​

Why is the cache being written (cache_creation_input_tokens is large) but never read (cache_read_input_tokens is 0)?

​

What is the minimum number of tokens needed for caching?

​

How long does the cache last? Can it be changed to 1 hour?

​

How do I enable caching in Dify / Cherry Studio?

​

Support Across Different Models

• Whether Prompt Caching is supported depends on the model itself.
• If the model inherently supports caching without requiring explicit parameter declarations, it can be supported through OpenAI-compatible forwarding.
• OpenAI supports prompt caching by default, applied automatically (prefix of 1,024 tokens or more). On models before GPT-5.6, cache writes have no additional fee and caches are cleared after 5-10 minutes of inactivity; on GPT-5.6 and later, cache writes are billed at 1.25x the input rate, cache reads at 0.1x, caches are retained for at least 30 minutes, and explicit cache breakpoints are supported. See GPT Prompt Caching.
• Claude requires the native cache_control: { type: "ephemeral" } declaration. Caching rate is 1.25 times the standard input cost (5-minute) or 2 times (1-hour), cached token retrieval costs 0.1 times the normal rate, with a 5-minute or 1-hour lifecycle. Details
• Deepseek V3 and R1 natively support caching. Caching rate equals the standard input cost, cached token retrieval costs 0.1 times the normal rate. Details
• Gemini implicit caching support:
  • Implicit Caching: Enabled by default for all Gemini 2.5 models. If your request hits the cache, cost savings are automatically applied. This feature is effective as of May 8, 2025. The minimum input token count for context caching is 1,024 for Gemini 2.5 Flash and 2,048 for Gemini 2.5 Pro.
  • Tips to improve implicit cache hit rate:
    • Try placing large, frequently reused content at the beginning of the prompt.
    • Try sending requests with similar prefixes within a short time window.
  • You can view the number of cache-hit tokens in the usage_metadata field of the response object.
  • Cost savings are calculated based on prefilled cache hits. Only prefill cache and YouTube video preprocessing cache are eligible for implicit caching.