What does a million-token context window actually buy you?
AI Architect
Key takeaways
- Needle-in-a-haystack scores measure recall of a distinctive fact rather than reasoning over a full context.
- Accuracy typically degrades for information placed mid-context, the "lost in the middle" effect.
- Cost and latency scale with tokens sent, so filling a large window on every request is rarely economical.
- Retrieval and long context are complementary: retrieve to select, then use the wide window to avoid aggressive chunking.
Every context-window expansion is announced as the end of retrieval, and it never is. The useful question is not how many tokens fit, but how accuracy, latency and cost behave as you approach that limit.
Does accuracy hold across the whole window?
No, and this is the finding that matters most. Advertised context length tells you how much fits, and says nothing about how well any of it is used.
Two lines of evidence. Research on positional sensitivity, Liu et al., found that models recall information at the start and end of a long context more reliably than material in the middle, and evaluations built to test reasoning rather than recall, such as RULER, consistently show effective context well short of the advertised maximum.
Where recall is reliable inside a long context
Show as text
| # | Layer | Note |
|---|---|---|
| 1 | Opening of the context | Most reliably recalled. Strongest evidence here. |
| 2 | Middle of the context | Recall degrades; the "lost in the middle" region |
| 3 | Close of the context | Second most reliable. Next-strongest here. |
The practical reading is that a window has a shape. Position within the prompt is an engineering variable you control, and putting critical material at the extremes rather than the middle is close to free.
Two habits follow from that. Order retrieved passages deliberately, strongest first and next-strongest last, rather than in the descending-similarity order your vector store returns. Left alone, that default puts your second-best evidence in the weakest position. Then measure your own effective window: plant a fact your system must use at 25%, 50% and 75% depth, and watch whether answer quality moves. If it does, your usable context is shorter than the number on the pricing page.
Why do benchmark numbers look better than your app?
Because the standard test is easier than your task.
Needle-in-a-haystack asks a model to locate one distinctive planted sentence in a large body of filler. That is single-fact recall over a low-noise haystack. Real work usually requires something harder:
- aggregating evidence across several passages that each contain part of the answer
- distinguishing between near-duplicate passages that disagree
- noticing that the answer is absent rather than confabulating one
# Measure YOUR effective window instead of trusting the advertised one.
# Plant a fact the system must use at several depths and watch quality move.
def probe_effective_window(client, filler, needle, question, depths=(0.25, 0.5, 0.75)):
results = {}
for d in depths:
cut = int(len(filler) * d)
prompt = filler[:cut] + needle + filler[cut:]
answer = client.ask(prompt, question)
results[d] = answer.is_correct # use your own grader here
return results # a dip at 0.5 means your usable window < the number on the boxEach of those degrades faster with length than recall does. A high needle score is necessary but nowhere near sufficient.
The mismatch is also one of noise. Benchmark filler is usually unrelated text, so the planted sentence is the only plausible answer anywhere in the window. Your corpus looks nothing like that. It holds dozens of passages that all seem relevant, several of them stale, near-duplicated or quietly contradicting each other. The two settings measure performance on completely different distributions, which is how a vendor’s chart and your own evaluation set can both be honest and still disagree.
# Position is a variable you control. Put the strongest evidence at the
# extremes rather than wherever the vector store happened to rank it.
def order_for_attention(passages):
ranked = sorted(passages, key=lambda p: p.score, reverse=True)
head, tail, middle = [], [], []
for i, p in enumerate(ranked):
(head if i == 0 else tail if i == 1 else middle).append(p)
return head + middle + tail # best first, second-best lastHow do the economics compare?
Relative cost per request, retrieval versus filling the window
Show data
| Item | Value (x) | Note |
|---|---|---|
| Retrieval-first (a few thousand tokens) | 1 | Fixed cost per query, roughly independent of corpus size. |
| Fill a 200k window | 40 | Scales with context length on every single call. |
This is where the argument is usually settled, and it rarely favours filling the window.
| Retrieval-first | Fill the window | |
|---|---|---|
| Tokens per request | Thousands | Hundreds of thousands |
| Latency | Low, dominated by search | High, scales with input |
| Cost per request | Low | 10–100× higher |
| Corpus size limit | Unbounded | Hard window ceiling |
| Failure mode | Wrong passages retrieved | Relevant passage ignored mid-context |
Latency is the line teams underestimate. Input tokens are processed before the first output token appears, so a 200,000-token prompt adds seconds to time-to-first-token on every request, whether or not the model needed all of it. For anything interactive that is felt directly, and streaming does not hide it.
Cost compounds differently too. Retrieval spends a small fixed amount per query almost regardless of corpus size, while filling the window spends in proportion to context length on every single call, so the gap widens with traffic rather than with the size of your data.
Provider guidance agrees on the practical fix. Anthropic’s long-context tips recommend putting the material you most need used near the top of the prompt.
Prompt caching changes this calculation when the same large context is reused across many requests, which is the one case where filling the window is clearly correct. For varied queries against a large corpus, retrieval stays ahead on every axis.
What is long context actually good for?
Three things, concretely:
- Whole documents that fit. A 200-page contract or an entire codebase module can go in intact, and you skip chunking, which is where most retrieval pipelines lose the semantic thread.
- Less destructive chunking. Retrieve at the document or section level instead of splitting into 500-token fragments that sever context.
- Multi-turn sessions with accumulated state. Long agent trajectories where the history itself is the context, including tool results arriving over MCP.
Notice what unites the three: in each, the boundaries of the content matter more than its volume. Long context is a chunking-avoidance tool. It does not avoid search.
The anti-pattern is using it as a substitute for deciding. Concatenating everything and hoping the model sorts it out reliably produces worse answers than a mediocre retriever. You have swapped a ranking problem you can measure and improve for an attention problem you can do neither with.
The right architecture for most systems is both: retrieve to decide what the model reads, and use the wide window so that what it reads arrives whole.
Do this
Measure your own effective context window
Advertised length is a capacity number. This is how to find the length your system can actually use, in an afternoon.
Build a fixed set of 30 to 50 real questions
Pull them from your own traffic, include cases you currently fail, and write down the correct answer for each. A set you already pass measures nothing.
Pick one fact each question depends on and plant it at three depths
Insert the needed passage at roughly 25%, 50% and 75% through the filler, keeping everything else identical. Position is the only variable you are changing.
Score the answers the way your product is judged
Exact-match will under-report. Use a rubric or a human pass, because the failure you care about is a plausible answer built on the wrong passage.
Plot accuracy against depth and find where it dips
A clear drop at the midpoint is the lost-in-the-middle effect showing up in your workload. The depth where quality falls is your real ceiling.
Reorder retrieval so the strongest evidence sits at the edges
Put the best passage first and the second best last. This costs nothing and recovers most of what the midpoint dip takes away.
Re-run the set on every retrieval or model change
Store the result with the version that produced it. Without that history a quality regression looks like noise.
Frequently asked questions
- Does a long context window make RAG obsolete?
- No. Retrieval decides which of your millions of tokens are relevant; the context window decides how many of those can be read at once. A corpus larger than the window still needs selection, and selection is cheaper than reading everything.
- What is the "lost in the middle" effect?
- The observed tendency for models to recall information at the beginning and end of a long context more reliably than information in the middle. It means position within the prompt is a real variable that you control.
- Why do needle-in-a-haystack results look so good then?
- Because finding one distinctive sentence is a much easier task than synthesising across many passages. The benchmark measures retrieval of a planted fact; most real tasks require aggregation, which degrades faster.
- What should I actually do with a large window?
- Stop chunking documents that fit comfortably inside it, keep retrieved passages whole rather than fragmenting them, and put the most important material near the start or end of the prompt.