Context Rot, Why Long Contexts Degrade and Curation Wins
Pull-quote: “A long context window is a storage spec, not an attention guarantee. What you can fit and what the model can use are different numbers, and the gap between them is where agents quietly get worse.”
Why this matters
Context windows grew a thousandfold and teams responded the obvious way: stuff everything in. The whole document set, the full conversation history, every tool output since the session began. The result is a failure mode with a name that stuck, context rot: model quality degrades as context grows, long before the window is technically full. The window is a capacity limit. Attention is the scarce resource inside it, and attention does not scale with capacity.
How rot actually happens
| Symptom | Mechanism | Curation counter |
|---|---|---|
| Mid-document facts get ignored | Attention favors the edges of long contexts | Retrieve narrowly; place critical facts at the boundaries |
| Confident answer, wrong passage | Near-miss distractors outrank the ground truth | Rerank hard, send the few passages that survive |
| Agent repeats an early mistake all session | Poisoned state carried forward turn after turn | Validate and compact state; restart from a clean summary |
| Quality sags as the session ages | Accumulation without curation | Working-set discipline with budgets per section |
| Costs climb while answers get worse | Every appended token bills and dilutes at once | Fetch on demand instead of preloading |
Two of these deserve emphasis. Distractor sensitivity means irrelevant context is not neutral filler; passages that look similar to the answer actively pull the model away from it, so retrieval precision matters more than retrieval volume. And context poisoning is the agent-specific variant: one hallucinated tool result or stale observation, appended to state and re-read every turn, steers the whole remaining session. Rot compounds in loops.
The two loops
hoarding agent curated agent
────────────── ─────────────
turn N: append everything turn N: retrieve on demand
│ history grows unbounded │ working set, budgeted
▼ ▼
attention spreads thin, compact on cadence: summarize
distractors accumulate progress, pin the invariants
│ │
▼ ▼
turn N+30: context is an turn N+30: context is a
archive, mostly noise briefing, mostly signalThe curation playbook
- Retrieve, do not preload. Fetch what the current step needs instead of front-loading the corpus. The context should hold the working set, not the warehouse.
- Rank and cut hard. Hybrid retrieval to find candidates, a reranker to order them, and a strict cut before the prompt boundary. Sending fewer, better passages beats sending more.
- Compact on a cadence. Summarize progress into structured state and discard the raw transcript. Pin the invariants, the goal, the constraints, the decisions already made, so they survive every compaction.
- Isolate subtasks. Give a sub-agent a fresh, scoped context and take back a structured summary, not a transcript. Isolation is how one polluted workstream is kept from poisoning the rest.
- Evaluate at operating length. A prompt that scores well at four thousand tokens can fail at eighty thousand. Run the eval suite at the context sizes production actually sees, or the rot ships silently.
Curation buys four things at once: accuracy, because distractors are gone; cost and latency, because tokens are fewer; cache efficiency, because a stable prefix with a small volatile tail is exactly what prompt caching rewards; and auditability, because a curated context is a record of what the model was actually shown.
Where curation is non-negotiable
Safety-critical retrieval is the clearest case. A system answering questions over a safety corpus should brief the model with a small set of ranked, cited passages rather than raw manuals, because a safety answer traced to the wrong document is worse than no answer. Multi-agent analysis workloads show the isolation rule at full strength: each agent works a scoped brief and returns a structured summary, so the coordinating layer never inherits a raw transcript or the rot it carries.
Closing
Long windows are genuinely useful; the mistake is treating them as an alternative to curation instead of more room to curate within. Retrieve narrowly, rank hard, compact on a cadence, isolate subtasks, and test at real lengths. The context is the model’s entire world for that call. Keep the world small and true, and the answers follow.