Signal, context

Bigger Context Windows Are Not Free

Huge windows tempt you to paste in everything. Curation beats capacity almost every time.

The Curio Team · 3 min readWritten with AI from the sources below and checked against them before publishing.

Context windows exploded. A few years ago a model might hold a couple thousand tokens. Now we routinely see windows of two hundred thousand tokens, and some stretch past a million. That sounds like a free upgrade. We get the temptation to paste in everything you own. But we have watched this backfire enough to say it plainly: a bigger window is a bigger container, not a smarter reader.¹

Here is the part that surprises people. Models do not attend evenly across a long input. Information at the very start and the very end gets used well, while material buried in the middle tends to fade. Researchers named this the lost in the middle effect, and it shows up across model families. So the position of a fact inside your prompt changes how likely the model is to actually use it. Placement is not cosmetic. It is leverage.²

Picture a fifty page contract pasted into one prompt. Ask about a clause on page thirty, and the model may whiff, not because it lacks the words, but because attention thinned out right where your answer lived. Now move that same clause to the top. The hit rate climbs. Nothing about the model changed, only the seating chart. We run this test on our own prompts constantly, and the pattern holds across tasks that need a specific detail rather than a broad summary.²

Retrieval beats stuffing almost every time we test it.

There is a second cost, and it lands on your bill and your latency. Every token you send gets processed, so a bloated prompt is slower and pricier on every single call. Send a hundred thousand tokens of background when three thousand would do, and you pay for the ninety seven thousand you did not need. Multiply that across a busy app. The waste compounds fast.³

So what actually fits? We think in terms of relevance, not capacity. Put the instruction and the question where the model reads best, near the top or the very bottom. Keep the genuinely load bearing evidence close by. Everything else is a candidate for the cutting room floor. When we curate ten strong passages instead of dumping two hundred mediocre ones, answers get sharper and cheaper at the same time.¹

A quick habit helps here. Before a call, we ask what the model must see to answer, and we throw out the rest. Summarize long history instead of replaying it verbatim. Strip boilerplate, dedupe near copies, and drop stale turns that no longer matter. If a document is only relevant in part, send the part. These edits feel small one at a time, yet together they routinely cut a prompt by half with no loss in quality, and often a gain.³

None of this means the big windows are useless. They are wonderful when the task genuinely spans everything at once, like tracing one thread through a long transcript, or when you cannot know in advance which passage matters. We reach for the full window then without guilt. The mistake is making it the default. Reach for it as the exception, and you keep both speed and accuracy on your side.

This is why retrieval still matters in the long context era. Pulling the right few chunks and handing over just those beats stuffing the whole corpus, and it beats it by a wide margin in our experience. Long windows are a safety net for when you truly cannot pre filter. They are not a strategy. Treat the window as scarce even when the spec says it is huge, and your results will thank you. Discipline with what you leave out is the real skill now, more than access to a giant window.