Context engineering: treat the window as a budget you spend
Pull-quote: “Truncation does not raise an error. It quietly deletes the constraint that mattered, then answers with confidence.”
Context engineering is budget management: the window is a fixed allocation, several claimants compete for it, and the ones you do not allocate deliberately will take space from the ones that decide the answer. Systems that treat the window as a budget degrade gracefully when a task grows. Systems that treat it as a place to append fail silently, which is worse, because a silent failure produces a confident answer built on material that was dropped before the model ever saw it.
This is the third post in a ten-part course on LLM harnessing. It follows prompt looping and designing a loop that closes, and it matters most for loops, because a loop spends context on every iteration. A single call that fits comfortably becomes a step 15 that does not.
What you will be able to do
- Name the seven claimants on your context window and allocate a budget to each.
- Explain how naive history accumulation causes quality collapse with no error message.
- Order the window so decisive material sits where it is used most reliably.
- Choose between summarisation, extraction, schema pruning, and reference by id.
- Design an eviction policy with pinned, compressible, and disposable tiers.
- Audit your own context spend and report the ratio of tokens spent to tokens used.
What competes for space in the context window?
Seven claimants compete for every window: system instructions, tool schemas, durable memory, conversation and step history, retrieved evidence, the scratchpad or plan state, and the reservation for the model’s own output. Allocate all seven or the unallocated ones will crowd out the rest.
Here is a worked allocation for a working ceiling of 64,000 tokens on a model whose advertised limit is larger. Adapt the shares to your workload; the point is that each line is a decision.
| Claimant | Budget | Why that size |
|---|---|---|
| System instructions and policy | 2,000 | Fixed cost, resent on every step |
| Tool schemas | 4,000 | Only the tools this step can call |
| Durable memory | 3,000 | Facts and skills, retrieved rather than dumped |
| Conversation and step history | 16,000 | Compressed, with older turns summarised |
| Retrieved evidence | 24,000 | The largest line, and the first to overrun |
| Scratchpad and plan state | 5,000 | Structured, rewritten in place each step |
| Output reservation | 8,000 | Never spent on input, or generation truncates |
Two decisions in that table do the heavy lifting. The working ceiling sits below the model’s advertised limit, so one oversized retrieval cannot push a step over the edge and take the run with it. And the output reservation is a real line, not slack, because a request that fits but leaves no room to generate fails in a way that looks like the model losing the thread.
Why does naive history accumulation fail?
Append-everything history fails because the input grows monotonically while the share of it that decides the answer shrinks, and the eventual overrun is usually handled by something that does not tell you. Cost and latency rise every step, and the moment you cross the limit, one of three things happens: the provider returns an error, which is the best case because it is visible; your framework drops the oldest messages, which can remove the system instructions or the original goal; or a middleware summarises without recording what it discarded.
The observable symptom is not an error. It is an agent that stopped honouring a constraint it was given at step three, or one that repeats work it already did, or one that asks for information it was told at the start. Those look like reasoning failures in a review and are context assembly failures in fact.
Tool results are the usual culprit, not the conversation. A single tool call can return several thousand tokens of JSON, and a loop that appends raw results verbatim reaches its ceiling in a handful of steps. Truncate at the boundary, keep a short digest in the window, and store the full result where the loop can fetch it again by id.
The fix is structural. Count tokens during assembly, compare against the budget before the call, and raise when a pinned item cannot fit. Log the assembled size on every step, in the form ctx=41208/64000, and alert when a step passes ninety percent of the ceiling. An overrun should be an incident with a stack trace, not a slow decline nobody can date.
How should you order what goes into the window?
Put the stable material first, background in the middle, and the decisive material last, next to the instruction it decides. Two independent mechanisms point the same way, which is unusual enough to be worth exploiting.
The first mechanism is retrieval reliability inside the prompt. Material at the beginning and the end of a long context is used more reliably than material in the middle, an effect that Liu and colleagues named lost in the middle in 2023. Long contexts remain useful; the lesson is to stop burying the paragraph that answers the question at position forty of sixty chunks.
The second mechanism is prompt caching, which rewards a byte-identical prefix. Anything volatile early in the prompt invalidates the cache for everything after it, so a timestamp in the system message or a tool list in random order is expensive twice: once in cache misses and once in the cognitive noise it adds.
A working order: system instructions and policy, then the tool schemas this step needs, then durable memory, then compressed history, then retrieved evidence with the best-scoring item last, then the scratchpad and plan state, then the current instruction. Note the detail in the evidence line. Rerankers usually emit the best match first, and the best match belongs closest to the instruction, so reverse the list before you pack it.
Which compression and eviction policies keep the decision intact?
Compression is how you fit more meaning into fewer tokens, eviction is how you decide what leaves when the budget is exceeded, and both must be policies in code rather than side effects of a library default. Four compression techniques cover most cases.
| Technique | What it keeps | What it loses | Use it when |
|---|---|---|---|
| Summarisation checkpoint | Narrative and decisions taken | Exact wording, and numbers unless pinned | History is long and mostly settled |
| Structured state extraction | Named fields the loop needs | Everything outside the schema | The loop compares facts, not prose |
| Prune unused tool schemas | The tools callable this step | Ability to call the rest until reloaded | The tool surface exceeds the step’s needs |
| Reference by id | A pointer plus a short digest | Inline detail until fetched again | Large documents and tool outputs |
Prefer extraction over summarisation for anything the loop must compare or check. Summaries drop numbers, identifiers, and thresholds first, because those read as detail rather than as meaning, and a loop that lost the threshold cannot verify against it. Pin identifiers, limits, and constraints verbatim in the scratchpad and summarise the story around them.
Eviction needs three parts: tiers, an order, and a rule that fails loudly. Pinned items include the system instructions, the goal, active constraints, and the current instruction, and they are never evicted. Refreshable items, such as retrieved evidence and tool schemas, can be dropped because you can fetch them again. Compressible items, mainly history, get summarised before they get dropped. Disposable items, such as raw tool output already digested, go first. When eviction reaches a pinned item, the assembly raises instead of quietly shrinking the prompt, which is what converts silent truncation into a visible failure.
Is retrieval a separate feature or part of context engineering?
Retrieval is part of context engineering, because the retriever decides how much of the window gets spent, on what, and in what order. It is the throttle on your largest budget line, implemented as a search system, so treat its parameters as budget parameters.
Set the number of results from the budget rather than from a default. If the evidence line is 24,000 tokens and your chunks run about 600 tokens, the arithmetic ceiling is 40 chunks, and with reranking you probably want a small fraction of that. Reranking is a budget technique before it is a quality technique: it lets you spend fewer tokens on the same decisive evidence, which frees space for history and scratchpad. Chunk size is the granularity of the spend, so oversized chunks buy irrelevant text alongside the relevant sentence.
How do you audit your own context spend?
Audit context spend by instrumenting assembly per claimant, tagging every item with an id, then measuring which ids the output actually used. The ratio of tokens spent to tokens used is the number that changes behaviour, and almost nobody measures it.
- Emit one line per step with the token count for each claimant and the total against the ceiling.
- Tag each item with an id as it enters the window:
mem:14,ev:e91,hist:s3. - Detect use. Require the model to cite the ids it relied on, and match tool arguments back to the item that supplied the value.
- Compute used over spent per claimant, per step, and keep it in the trajectory.
- Rank claimants by tokens spent with no observed use, and cut there first.
- Track cost per useful token over time. The trend matters, not the absolute value.
A per-step line looks like this, and the components add up to the total, which is worth asserting in a test:
step 7 ctx=41208/64000 sys=2010 tools=3940 mem=2880 hist=15600 ev=12778 out=4000
ev_used=3/21 progress=2 unresolved verify=passYou know the audit is working when a line like ev_used=3/21 makes you uncomfortable. Eighteen retrieved chunks were paid for, sent, and ignored, which is a retrieval problem, a reranking problem, or a chunk size problem, and now it is visible enough to fix. Run this on a week of trajectories and the ranking of what to cut writes itself.
Does prompt caching change the budget?
Prompt caching changes the price of a token, not the number of tokens, so it lowers the cost of a budget without loosening it. Providers key their caches on an exact prefix, which means a hit requires everything before the changed content to be byte-identical, and a single volatile early element invalidates the rest.
Three practices follow. Keep the prefix stable: fixed system instructions, tool schemas in a deterministic order, no timestamps or session ids near the top. Put volatile content late, which is the same advice the ordering section gives for a different reason. And batch changes to early content rather than editing it every step, because each edit pays for a fresh cache entry.
One caution. Cheap tokens tempt you into keeping content you would otherwise drop, and irrelevant context still costs you selection quality even when it costs almost nothing in dollars. The used-to-spent ratio remains the check, because a cached distraction is still a distraction.
Where this goes wrong
Silent truncation. The symptom is a quality cliff partway through longer runs, with no error anywhere. The cause is a library dropping the oldest messages when the request exceeds the limit. Add budget-checked assembly with pinned tiers, and raise on overrun so the failure has a timestamp and a stack trace.
Summary drift. The symptom is an agent that violates a numeric constraint it demonstrably knew earlier. The cause is a summarisation checkpoint that kept the narrative and dropped the number. Extract constraints into structured state, pin them verbatim, and let the prose around them compress.
Retrieval flood. The symptom is cost per run climbing while answers do not improve. The cause is a default result count and no reranking, so the evidence line consumes budget that history and scratchpad needed. Derive the result count from the budget, rerank, and watch the used-to-spent ratio per step.
Cache thrash. The symptom is caching enabled and spend unchanged. The cause is volatile content early in the prompt: a timestamp, an unordered tool list, a session identifier. Freeze the prefix, move anything that changes per step to the end, and verify with the provider’s reported cache hits.
Common questions
Is a bigger context window the fix?
No, it buys headroom rather than discipline. A larger window still costs more per step, still uses material in the middle less reliably than at the edges, and still forces the same eviction decision one task size later. Without budget discipline, a bigger window mostly makes the failure more expensive.
How do I count tokens before I make the call?
Use the tokeniser for the model you are calling, count the fully assembled string, and treat counting as part of assembly rather than a debugging step. Rough estimates by character count drift badly for code, JSON, and non-English text, which are exactly the payloads that overrun.
Should I summarise history or extract it?
Extract anything the loop compares, checks, or verifies against, and summarise the narrative around it. Extraction produces named fields you can assert on, while a summary is prose that may or may not still contain the threshold you need. Many production loops do both: a structured state object plus a running summary.
Where does agent memory fit in the budget?
Memory is a claimant with a read budget, not an exception to the budget. Retrieve from memory rather than loading it, cap what any single retrieval may contribute, and attach provenance so you can audit which memory entries actually changed a decision. The write side is a separate design problem covered next in this course.
What ratio of spent to used should I target?
There is no universal figure, and anyone quoting one is describing their own workload. Measure your ratio, rank the claimants by wasted tokens, cut the worst one, and watch the trend across a week of trajectories. The value of the metric is the ranking it produces, not the number itself.
Next steps
Read agent memory architecture next, because memory is the claimant with the strongest tendency to grow, and its write policy decides how much context you spend on every later run.
For cost mechanics beyond context, cost and latency engineering for LLM systems covers routing and caching in depth, and memory and skills in Hermes Agent shows a harness with a compression command and a documented context floor.
Then audit one workload this week. Instrument assembly per claimant, tag items with ids, and produce one ev_used figure for a day of runs. That budget discipline is the retrieval practice behind EvidAI in pharmaceutical evidence review, where the evidence line is the largest one and gets audited like any other spend.