Cache and Cost

Why this is the first Production module

Most early API systems are correct but expensive. The same system, with prompt caching applied properly, often runs at 10–25% of the uncached cost. Nothing else in the Production tier compounds across the rest of the axis the way this does. Authentication, model selection, even tool design — fine, fix later. Caching is the one that pays back on day one.

What the cache actually is

Every API call sends a sequence of tokens: system prompt, tool definitions, conversation history, the new user message. The model reads the whole sequence top to bottom and generates a response. If the first 4,000 tokens are identical to a call you made five minutes ago — same system prompt, same tools — the model still has to attend to all of them. You still pay for them.

Prompt caching lets you mark a point in the sequence and say "everything before this is stable; re-use the work you did last time." On a cache hit, the input tokens before that mark are billed at roughly 10% of the normal input rate. On a cache miss (first time, or because the cached content expired), the same tokens are billed at slightly more than the normal rate — the write cost. After one or two hits, the write has paid for itself.

The cache is content-addressed: same bytes, same hit. Any change before the breakpoint busts the cache. Anything after the breakpoint is uncached and billed normally.

How to mark a breakpoint

You add a cache_control field to the content block where the stable region ends. The SDKs use the same shape; here it's in Python:

response = client.messages.create(
    model="claude-sonnet-4-6",
    max_tokens=1024,
    system=[
        {
            "type": "text",
            "text": LONG_SYSTEM_PROMPT,
            "cache_control": {"type": "ephemeral"},  # cache up to here
        }
    ],
    messages=[
        {"role": "user", "content": user_question}  # uncached, varies per call
    ],
)

The system prompt is cached. The user message changes on every call and is sent uncached. On the second call with the same system prompt, the API recognises the prefix, reuses the cached state and bills the system tokens at the cached rate.

You can place breakpoints in tool definitions and in conversation history too. The typical pattern in production:

system    → cache_control here (large, stable)
tools     → cache_control here (large-ish, stable across runs)
messages  → conversation history; cache_control on the last assistant
            turn if you're maintaining a long agent loop
final user message → uncached

Multiple breakpoints are fine. The cache is hierarchical; later breakpoints depend on earlier ones still being valid. Read the SDK docs for the current limit on breakpoints per call.

Reading the usage block

The usage field on every response tells you what happened:

{
  "input_tokens": 27,                     # uncached input
  "cache_creation_input_tokens": 4123,    # first time, you paid to write the cache
  "cache_read_input_tokens": 0,           # nothing read from cache yet
  "output_tokens": 312
}

That's the first call. The next call with the same prefix:

{
  "input_tokens": 31,                     # the new user message
  "cache_creation_input_tokens": 0,       # didn't write anything new
  "cache_read_input_tokens": 4123,        # full prefix served from cache
  "output_tokens": 287
}

That's a hit. The 4,123 tokens that would have cost the standard rate now cost roughly a tenth. If you make 100 calls before the cache expires, you paid the write cost once and the read cost 99 times. The arithmetic is decisive.

The decision

Not every workload benefits. Caching pays back when (a) the prefix is large enough that the savings matter and (b) you'll hit the same prefix more than twice before it expires.

Decision · cache yes or no

Should you cache this prompt?
├─ Is the stable prefix > 1024 tokens?         no  → don't bother
│                                                yes
│                                                ↓
├─ Will the same prefix be reused > 2 times    no  → write cost > savings
│  within the cache TTL?                         yes
│                                                ↓
├─ Is the stable prefix actually stable?       no  → cache will bust;
│  (system + tools + standing instructions)        you'll write every time
│                                                yes
│                                                ↓
└─ Cache it. Read the usage block to confirm.

The cache-bust hygiene rule

The cache breaks on any change to the bytes before the breakpoint. The two failure modes are dynamic data sneaking into the system prompt and tool definitions reordering between calls.

Don't templated the system prompt with per-request values. A system prompt that contains today's date, the user's name or the request ID will produce a different cache key on every call. Move those values into the user message instead. Keep the cached region truly stable.

Stabilise tool order. If you build the tool list dynamically — say, from a set of MCP servers — the order may change between calls. Sort it. Same set, same order, same cache key.

Both bugs look identical in the wild: usage shows cache_creation_input_tokens on every call and cache_read_input_tokens stays at zero. If you see that, something in the supposedly-stable region is moving.

The five-minute test

Take the loop from API-02. Add a 1,000-token system prompt (paste a reference document, a coding style guide, anything). Make ten calls in a row with that system prompt and different user messages. Print the usage on every call.

Now add cache_control to the system prompt and run the same ten calls. Compare. The first call costs slightly more (write cost). The next nine cost dramatically less. The total cost across the ten calls is the test.

The exact savings depend on the prefix length and the model. Sonnet 4.6 with a 4,000-token cached prefix and ten calls runs at roughly a third the uncached cost — including the first-call write. The numbers are big enough that this is the optimisation to know before you ship.

Next

This is the last module in phase 1 of the Build axis. From here the Use axis is still where most of the value lives if you're a working human. The Build axis fills out in phase 2 with deeper Production modules (rate limits, model selection at volume, multi-environment) and the Agents tier (Skills, sub-agents, memory). For now, go back to the course index or pick up where you left off on the Use axis.