Lost in the middle is architectural. Reordering your prompt won't fix it.
A March 2026 paper finds the U-shaped curve in untrained transformers, before a single training step. That changes which lever is worth pulling.
"Lost in the middle" is the paper people cite when their agent forgets an instruction it followed ten minutes ago. The citation is right and the conclusion drawn from it is usually wrong. The bias is not a quirk of 2023 models that better training will iron out. One 2026 paper puts it in the architecture, present before the first gradient step. That makes it permanent, and it makes prompt reordering the wrong response to it.
I went looking because I had just measured my own Claude Code window: 43k of MCP tool schemas sitting in front of every question I typed, out of 99.1k total. I wanted to know whether being buried behind that payload was costing me accuracy, and whether moving things around would buy it back. The answer to the first is yes. The answer to the second is no, and the reason is more interesting than the U curve everyone quotes.
The 2023 numbers are worse than the version you've heard
Lost in the Middle: How Language Models Use Long Contexts, by Nelson F. Liu and six co-authors, tested multi-document question answering and synthetic key-value retrieval. The multi-document setup is the one worth understanding. The model gets 20 documents, exactly one of which contains the answer, and the other 19 are retrieved, on topic, and wrong. The experiment slides the right one along the list and reads the accuracy off.
| where the answer document sits | GPT-3.5-Turbo accuracy |
|---|---|
| first of 20 | 75.8% |
| ninth of 20 | 53.8% |
| twentieth of 20 | 63.2% |
| no documents at all (closed book) | 56.1% |
| the answer document alone (oracle) | 88.3% |
Everyone quotes the 22 point spread. The line worth stopping on is the fourth. Handed the correct document in the middle of a stack, the model did worse than being handed nothing and made to answer from memory. The right information had negative value.
The paper also killed the obvious workaround before anyone proposed it. GPT-3.5-Turbo at 4k and the 16k variant traced nearly identical curves on the same document counts, and Claude-1.3 against Claude-1.3-100k behaved the same way. A bigger window was not a better window.
The shape shows up before the model has learned anything
The comforting reading of that paper is that 2023 models were bad at long contexts and 2026 models are better. Lost in the Middle at Birth: An Exact Theory of Transformer Position Bias, posted by Borun D Chowdhury in March 2026, argues the shape was never learned in the first place.
The claim: the U-shape is present at initialization, before any training, and before positional encoding contributes anything. It falls out of the geometry of a causal decoder with residual connections. The paper models attention as iterated Cesàro matrix powers and names three regions: a logarithmic primacy tail at the start from causal masking, an O(1) recency anchor at the final token from the residual stream, and in between a dead zone decaying like O(1/(H-1)!) in depth H.
Factorial decay in the middle. Not a tuning problem.
The empirical half is the part that made me sit up: untrained Qwen2 and GPT-2 architectures show the curve at initialization, with or without RoPE, and the pattern is still there after standard pretraining.
Hold this one loosely. It is a single-author theory paper from four months ago, its empirical leg stands on untrained small models rather than frontier systems, and I have not seen it replicated. I am reporting it because if it is right it explains why two years of better long-context training did not delete the effect, not because a preprint settles anything.
The bias stayed. Its predictive value didn't.
Now the result that stops this from becoming a doom post.
Context Rot, by Kelly Hong, Anton Troynikov and Jeff Huber in July 2025, ran 18 models across four families: Claude Opus 4 and Sonnet 4 down to Haiku 3.5, o3 and the GPT-4.1 line, Gemini 2.5 Pro and Flash, three sizes of Qwen3. They varied the needle across 11 positions in needle-in-a-haystack and reported no notable variation between them.
Both things are true, and the resolution is the whole point of this post. An architectural bias in how attention distributes is not the same measurement as accuracy on a task. A current model can carry the U-shape in its attention and still saturate an easy retrieval task from any position, the way you can read a road sign in poor light and fail to read a paragraph in the same light. The bias sets how much margin you have. It does not, on its own, tell you when you run out.
So position is real, permanent, and no longer the variable you should be tuning. What eats the margin is everything else Chroma measured. A single distractor, topically right and factually wrong, pulled accuracy below baseline, and four compounded it. Low similarity between question and answer accelerated the decline. All 18 models did better on a shuffled haystack than on one that read as a coherent flow of ideas, which is odd enough that I would want it replicated before building on it. Their memory benchmark compared a focused prompt of roughly 300 tokens against the roughly 113k full version of the same task and found substantial gaps.
2026 moved the question into agent transcripts
The three papers that landed this year stopped asking about documents and started asking about sessions, which is what you and I actually run.
Monitoring, May 2026. Classifier Context Rot, by Sam Martin and Fabien Roger, had frontier models read coding agent transcripts and flag a subtly dangerous action. Opus 4.6, GPT 5.4 and Gemini 3.1 missed it 2x to 30x more often when it sat after 800K tokens of benign activity than when it was shown on its own. Their framing matters as much as the number: real agent transcripts routinely exceed 500K tokens while prior monitoring benchmarks rarely passed 100K, so evaluations that ignore length have been overstating how well this works.
Search, June 2026. Diagnosing and Mitigating Context Rot in Long-horizon Search, by Shijie Xia, Yikun Wang, Zhen Huang and Pengfei Liu, ran four flagship open-source models across three benchmarks and named a failure mode I recognise immediately: as context accumulates, models give up outright or commit early to an uncertain answer. Not confusion. Surrender.
Agents, February 2026. LOCA-bench, by Weihao Zeng, Yuzhen Huang and Junxian He, grows the environment under an agent and watches success rate fall as states get more complex. Their useful finding is the mitigation: context management strategies substantially improved overall success.
Three teams, three tasks, one variable. None of them recommends reordering anything.
Your session is the distractor experiment, not the haystack
Here is why I do not think a flat position curve lets anyone relax.
Chroma's needles sat in unrelated filler. Liu's sat among 19 documents retrieved for the same query, all on topic, all plausible, all wrong. Only one of those resembles hour three of a coding session.
By then your window holds the previous version of the function you just rewrote, two components with almost the same name, a deprecated helper that still compiles, a test describing behaviour from four commits ago, and an error you already fixed. None of it is filler. Every piece is topically perfect and currently wrong, which is the exact profile of the distractors that dragged accuracy below baseline on a single appearance. You are not accidentally running the easy benchmark. You are hand-building the hard one, and 800K tokens is not a hypothetical length for it.
The tool schemas do the same thing quietly. 43k in my case, 127 tools, describing capabilities I would not touch that session. Every one is a plausible action to weigh before reaching the thing I asked for.
What I can measure, and what I can't
I have not run a positional eval on my own sessions. Everything above is other people's measurement, and I would rather label it than let borrowed numbers read as mine.
What I can measure is the layout. /context tells me exactly how many tokens sit ahead of my question and what they are. That is a real number about my setup and it says nothing about accuracy. The honest experiment is one I have not built: the same task and fixtures, once in a clean session and once after loading four near-miss files, graded the way I graded model choice for a real feature.
The limits on the sources cut in both directions. Liu's headline numbers come from a model three generations retired. Chroma's flat position curve is one task family. Chowdhury's theory is a preprint with untrained models under it. Martin and Roger measure classification of transcripts rather than an agent doing the work, which is close to my situation but not identical to it.
So pull the lever that has papers under it
Repeating your instructions periodically through a long session is now the one prompt-shaped intervention with 2026 evidence behind it, and even there the finding is that it partially mitigates. Take the partial fix. Just notice it is a patch applied to a bias that a preprint says was there before the model saw its first token.
The rest is deletion, and it is duller than a prompting trick. Delete the file you opened on a hunch. Start a fresh session when the task changes instead of steering the old one across the boundary. Ask for the conclusion instead of the investigation, so what lands in the window is an answer rather than the search that produced it. Trim the tool schemas you never invoke. Every one of those removes a confident near-miss from the pile, which is the variable that showed up in all five papers.
More context was never the goal. Fewer plausible wrong answers is.
Open /context and read what is in your window right now. Count the files you would defend keeping if someone made you justify each one, then cut the rest before you write the next prompt. The remaining levers are in production LLM cost optimization.
Questions about this post.
- What is the lost in the middle problem in LLMs?
- It is the finding from Liu et al. 2023 that a model answers better when the relevant document sits at the start or end of its context than when it sits in the middle. On multi-document QA with 20 documents, GPT-3.5-Turbo scored 75.8% with the answer first, 63.2% with it last, and 53.8% with it in the middle. The shape of that curve is a U.
- Is lost in the middle still true in 2026?
- The bias is, and one 2026 paper argues it is permanent. Chowdhury's "Lost in the Middle at Birth" finds the U-shape in untrained Qwen2 and GPT-2 at initialization, before any training and with positional encodings switched off, and reports it surviving pretraining. What changed is its predictive value: Chroma tested 18 frontier models in 2025 and found no notable variation across 11 needle positions on needle-in-a-haystack.
- Does adding more context make an LLM perform worse?
- It can make it worse than adding nothing. In the 2023 paper, GPT-3.5-Turbo scored 53.8% with the answer document buried in the middle of 20 documents and 56.1% with no documents at all. The right answer was in the window and the model still did better without it.
- Should I repeat my instructions periodically in a long session?
- Yes, and that one now has 2026 evidence behind it. Martin and Roger found periodic reminders spread through a transcript partially mitigated long-context failure in monitoring classifiers. Partially is the operative word. It is a patch on a structural bias, not a substitute for keeping the transcript shorter.
- How much does long context hurt coding agents specifically?
- In May 2026, Martin and Roger measured Opus 4.6, GPT 5.4 and Gemini 3.1 classifying coding agent transcripts. All three missed a subtly dangerous action 2x to 30x more often when it sat after 800K tokens of benign activity than when it was presented alone. They also note real transcripts routinely pass 500K tokens while prior benchmarks rarely went past 100K.