Signal, cost

Freeze the Top of Your Prompt

Prompt caching reuses a stable prefix across calls, cutting input cost and latency for anyone who keeps hitting the same context.

The Curio Team · 3 min readWritten with AI from the sources below and checked against them before publishing.

Here's the idea in one line. Models charge you to read text. When your calls share a big chunk of context, a system prompt, a rulebook, a long document, you pay full price to reprocess it every single time. Prompt caching ends that waste, and the savings are not small.¹

The trick is order. Put the parts that stay the same at the front of your request, and the parts that change at the end. The model processes text left to right, so a matching prefix can be reused from an earlier call. That reused span is the cache. Change one character near the top and the cache breaks, so keep the steady stuff steady.

How much does it save? On Claude, reading from the cache costs roughly a tenth of the normal input price, though writing to the cache the first time runs a little above full price. So the first call is slightly pricier. Every call after it, over that same prefix, gets cheap fast. The default cache window is short, about five minutes, and each hit refreshes it.¹

Picture a real number. Say your system prompt and reference docs run 20,000 tokens, and the user's question adds 200. Without caching you pay to process 20,200 tokens on every call. With a warm cache you pay full freight on roughly 200 new tokens and a tenth of the rate on the rest. Do that a thousand times a day and the difference is real money, not a rounding error. We've watched this line item drop by more than half on our own workloads.¹

Pay once to read it. Pay a dime to reuse it.

Who wins here? Anyone making repeated calls over the same context. A support bot with a fixed policy prompt. A coding agent that ships the same repo files on every turn. A document tool answering ten questions about one contract. On OpenAI's API, caching kicks in automatically for long shared prefixes and discounts those repeated tokens by roughly half, so you often benefit without touching your code.²

How do you turn it on? It depends on the provider. Anthropic asks you to mark where the stable prefix ends, so you place a cache breakpoint after your fixed blocks. OpenAI does it for you once a prefix is long enough. Either way, the mental model is identical. Order your context from most stable to least, and let the boring, unchanging bytes live at the top where they can be reused. That's how we shape every agent we ship.^{1 2}

There's a second payoff beyond money. Cached tokens skip most of the heavy processing, so responses come back faster too. The design rule is simple. Stack your prompt so the durable material sits up top: instructions, schemas, reference docs, few-shot examples. Let the user's actual question sit last. We build our own agents this way, and it's the cheapest optimization we know.

Latency matters as much as cost for anything interactive. A chatbot that reprocesses a giant prompt on every message feels sluggish. Reuse that prefix and the first token arrives sooner, because the model already did the reading. Faster and cheaper rarely travel together. Here they do.

The most common mistake we see is small. People paste a timestamp or a session id into the top of the prompt. That one moving value invalidates the whole cache on every call, and the discount vanishes. Push volatile fields down. Keep the head frozen and the tail free.

One caveat keeps you honest. Caching only helps when the prefix is genuinely stable and long enough to clear the provider's minimum. Shuffle your system prompt every call and you get nothing. So freeze the top, vary the bottom, and let the meter reward you.