Why does the model ignore what's in the middle of the context?

Lost in the middle is the drop in performance when the relevant information sits in the middle of the input rather than at its edges. The name comes from a 2023 study that measured the effect with content held fixed and only position varying: accuracy is highest at the start and the end, and falls in the middle. It shows up even in models sold as long-context.

The result stings because it contradicts the intuition that a large window is uniform space. It is not. Where you put the passage matters, and it matters more than most of the wording tweaks a team tries before finding this out. That is why instruction order changes the result and why structuring a long prompt is an engineering decision rather than an aesthetic one.

The effect is one cause of the broader phenomenon of degradation as context accumulates. It is not the same problem, but it is the better measured of the two.

The U-shaped curve

The design of the experiment is what gives the result its force, so it is worth spelling out.

The authors built a multi-document question answering task: a real Google query taken from NaturalQuestions, and a set of Wikipedia passages in which exactly one contains the answer and the rest are distractors pulled by a retrieval system. Then they varied one thing — the position of the answering passage — holding everything else identical¹.

With 20 documents in context, GPT-3.5-Turbo lost more than 20 points between the best and the worst position. And here is the number that closes the argument: in the 20- and 30-document settings, performance at the worst position fell below the same model’s performance with no documents at all, which was 56.1%. Having the answer in the window, in the wrong place, was worse than not having it.

For scale, the same model given only the correct passage, with no distractors, reached 88.3%. The distance between 88.3% and something under 56.1% is the price of position stacked on the price of distractors.

Two further observations from the same work close off the easy exits. Extended-context models did no better than their non-extended counterparts when the input fit in both: the curves were nearly superimposed. And GPT-4, evaluated on a subset, scored higher in absolute terms than everything else and still drew the same U¹.

Only the position moves

It is worth pressing on this point, because it is what makes the result actionable.

It is not that the model has less information, nor that the context is longer, nor that the question is harder. The three rows in the figure hold the same twenty documents, the same token count and the same question. The only difference is the order in which the documents were concatenated.

In a retrieval system, that is a variable you control for free. The ranker already returns passages ordered by relevance, and the decision about where to place them in the window is usually made by accident — by the order in which a loop walks the list.

That is unusual among context problems. Most of them cost something to fix: smaller windows mean less information, compaction risks dropping what mattered, better retrieval means a new model to run. Reordering the same passages costs one line and no tokens. It is the cheapest lever in the whole discipline, and it is routinely left in whatever position the retrieval library happened to emit.

One architectural note that helps explain the mechanism: encoder-decoder models proved relatively robust to position, but only within the sequence length they saw during training. Evaluated on longer sequences, they draw the U as well¹. The models everybody uses today are decoder-only, and in those each token sees only what came before it — which makes position a structural factor rather than an implementation detail.

How it shows up in your application

The symptom rarely arrives labelled “position problem”. It arrives in three other shapes.

The right answer was there and the model did not use it. This is the most frustrating to debug, because the retrieval log shows the correct passage among the results. You check, the content is right, and you conclude the model is bad. It did read the passage, but it read it with little attention, because the passage sat fourth of six, in the middle of a window that already carried history.

Raising top-k made the system worse. Somebody goes from five passages to fifteen “to be safe” and the hit rate drops. That makes sense: the ten new ones mostly land in the middle, push what was good away from the edges, and what came in is near-relevant enough to compete.

The same case passes and fails depending on the day. If the number of retrieved documents varies per query, the relative position of the correct passage varies with it. What you get is an accuracy rate that oscillates with no code change, and a team convinced the provider changed the model.

All three are diagnosed with the same instrumentation: for every evaluated answer, log where in the window the passage that should have been used actually sat. Two weeks of logs answer whether your problem is positional, and it is one of the few context diagnoses that produces a clean chart.

Why it happens

The explanation is not single and the field has not settled. Three lines of evidence combine.

Initial tokens draw disproportionate attention. A 2023 paper described what it called an attention sink: the first tokens in the sequence receive high attention scores even when they are not semantically important. The authors showed this while investigating why sliding-window attention breaks once the first tokens leave the cache — keeping just those initial tokens recovers much of the performance². It is a plausible mechanism for the left arm of the curve.

A recency bias pulls the other way. A paper from the same year measured that relevant information located earlier in context receives less attention on average, and used that to propose reordering documents by the attention they receive, with the most attended placed last³. Put the two effects together and the middle has no advocate: it is neither start nor end.

The bias is positional, not semantic. The most direct measurement is from 2024. The authors connected lost in the middle to an intrinsic U-shaped attention bias: tokens at the beginning and the end receive more attention regardless of their relevance. From that they built a calibration mechanism that discounts the positional component, and reported gains of up to 15 percentage points over existing methods on retrieval-augmented tasks⁴.

One hypothesis the original work tested and rejected deserves a note, because it is the first that occurs to everyone: this is not the fault of instruction tuning. Since the instruction usually sits at the start of tuning data, it would be reasonable to assume the model learned to look there. But the base model, before any tuning, draws the U as well. Tuning only narrowed the gap between best and worst position, from about 10 points to something near 4¹.

And there is a scale dependency that is rarely cited. Evaluating the Llama-2 family at three sizes, the authors found the U shape only in the 13B and 70B models; the 7B was purely end-biased¹. Primacy bias looks like something that emerges with scale, while recency has been there all along.

What does not fix it

Retrieving more documents. It is the instinctive reaction and it makes things worse. In the same study, going from 20 to 50 retrieved documents improved GPT-3.5-Turbo by around 1.5% and Claude-1.3 by around 1%, while the context more than doubled¹. Performance saturates long before retrieval coverage does.

Repeating the question before and after the documents. This technique works spectacularly on one task and barely at all on the other. On synthetic key-value retrieval, models went to near-perfect, against a worst case of 45.6% without it. On multi-document questions it slightly improved the case where the information sits at the very start and slightly hurt the rest¹. Worth testing, not worth assuming.

Waiting for the next model. In 2025, a benchmark attacked a bias in earlier tests: when question and passage share the same words, the model solves it by literal matching and the test becomes too easy. With minimal lexical overlap, 11 of 13 models claiming at least 128,000 tokens fell below half their own short-context performance by 32,000. GPT-4o, one of the best in the batch, went from 99.3% to 69.7%⁵. The authors attribute the decline to the difficulty attention faces in long contexts when there is no literal match to lean on.

Trusting the needle-in-a-haystack test. A model can find the needle in a million tokens and fail at 32,000 when the task requires aggregating scattered information. That is what RULER measured across 17 models and 13 tasks: half of those advertising at least 32,000 tokens failed to hold acceptable performance at that length⁶. The simple retrieval test barely separates models, which is why it is the one that gets publicised.

Swapping the ranker and stopping there. A better reranker puts the right passage in first place more often, which genuinely helps. But it does not decide where first place lands inside the window, and the library default is “at the top of the document list”, which in a context carrying history can be exactly the middle of the sequence the model reads. Ranking and positioning are two decisions, and usually only one of them has an owner.

What to do

1. Cut before you sort. Five well-chosen passages beat twenty almost every time. The gain from lowering top-k is usually larger than the gain from any reordering.
2. Put the most likely candidates at the edges. If your ranker gives a score, put the top result at the end, right before the question, and the second at the start. The middle gets the rest.
3. Repeat the question after the documents. It is cheap, it helps on some tasks and it costs almost nothing in tokens. Measure before adopting it as a rule.
4. Delimit the blocks. A header with source and date on each passage helps the model know what it is reading and helps you audit where the answer came from.
5. Measure your own case with the three-position test. Take twenty real questions, build each with the answering passage at the start, the middle and the end, and compare accuracy. An afternoon of work gives you the size of the effect on your model, with your task.
6. Redo the measurement on every model change. The curve changes shape and depth between families, and your ordering was calibrated for the previous one.

Step 5 is what replaces the argument with evidence. The literature says the effect exists and gives the order of magnitude; it does not say what it costs in your pipeline, with your ranker and your model. That number is yours and nobody is going to measure it for you.

Footnotes

1. Liu et al. (2023) varied the position of the answering passage in multi-document QA and key-value retrieval, measuring the U-shaped curve, the drop below no-document performance, the absence of gains in extended-context models, and the persistence of the effect in base models. ↩ ↩² ↩³ ↩⁴ ↩⁵ ↩⁶ ↩⁷

2. Xiao et al. (2023) described the attention sink: the first tokens receive high attention scores despite carrying no semantic weight, and keeping them in cache restores sliding-window attention performance. ↩

3. Peysakhovich and Lerer (2023) measured that relevant information placed earlier receives less attention on average and proposed reordering documents by the attention they receive. ↩

4. Hsieh et al. (2024) connected the effect to a U-shaped attention bias independent of relevance and proposed a calibration that discounts the positional component, with gains of up to 15 percentage points on retrieval-augmented tasks. ↩

5. Modarressi et al. (2025) built tests with minimal lexical overlap between question and passage, forcing latent association, and measured large drops from 32,000 tokens onward in models claiming 128,000 or more. ↩

6. Hsieh et al. (2024) evaluated 17 models on 13 tasks, including multi-hop tracing and aggregation, and found sharp degradation well before the advertised limit. ↩