How does the harness decide what goes in the context window?

The harness rebuilds the context window from scratch on every iteration, following a priority order and a budget. System instructions, the tool catalogue and the project configuration always go in. The user’s request always goes in. History and tool output go in up to an occupancy threshold, and past that point the harness starts cutting, always taking the cheapest thing to recover.

None of this is the model’s choice. It receives an already assembled token sequence and has no idea what was left out. How to allocate the space in the window is the decision behind that assembly, and in an agent it runs dozens of times per task instead of once per prompt.

The difference between an agent that finishes and one that gets lost around iteration 15 usually lives here. Two levers do most of the work: pulling content in only when it’s needed and summarising what already happened once there’s no room left. The first avoids the problem; the second manages it once it exists.

The budget, split up

The numbers in the figure come from a mid-sized coding-agent task in a 200K-token window. They’re plausible, not universal: measure your own, which takes one line of code.

System instructions (9K). The agent’s baseline behaviour. Resent in full on every call and almost always larger than whoever wrote it assumes.

Tool catalogue (18K). Name, description and schema for every available tool. Also resent in full. It’s the part that grows fastest when someone connects new servers, and the one that benefits most from loading on demand.

Project configuration. Conventions, build commands, what not to touch. It rides along with the system block and deserves the same care: whatever lives in the project file is fixed cost on every iteration for the life of the repository.

Task (2K). What the person asked for. It’s the smallest segment in the figure and the only one that exists because of the user. That proportion is worth looking at once.

History (35K). Previous turns, with the tool calls and the reasoning between them. Grows slowly and predictably.

Tool output and files (96K). Whatever came back from each command and whatever was read from disk. Grows in steps rather than curves: one unbounded grep or one failing test suite adds 30K tokens at once.

Headroom (40K). The room that has to remain for the answer. A harness that fills the window to 100% gets an API error instead of a response, and the failure mode is worse than it sounds: it happens exactly when the task is furthest from the start.

Adding up, 27K tokens are fixed cost paid on every call before any work happens. Across a 30-iteration task that’s 810K tokens of prefix. Prompt caching removes most of that while the prefix stays stable, which is why the order in which the harness assembles the prompt has a financial consequence: anything variable placed before the catalogue invalidates the whole cache.

The priority order

Every harness has one, even when nobody wrote it down. From what leaves last to what leaves first:

1. System instructions. Never leave. Without them the agent changes behaviour mid-task.
2. The original request. Never leaves. It’s the only definition of success in the room.
3. Constraints given at any point. “Don’t touch migrations/” has to survive everything. It’s the item most often lost in practice, because it arrived as a chat message and gets treated as ordinary history.
4. Decisions already made. “We’re going with Postgres.” Losing this makes the agent reopen the argument twenty iterations later.
5. The most recent turns. The last two or three, verbatim.
6. Older turns. Candidates for summarisation.
7. Old tool output. First to go, because you can nearly always run the command again.

Items 3 and 4 separate a good harness from a bad one. They have the shape of conversation and the weight of instruction, and a naive harness goes by shape. A constraint typed in message two is, structurally, one more user turn among thirty. Nothing in the transcript marks it as binding, so it ages out on exactly the same schedule as a question about where a file lives. The fix is unglamorous: a separate slot that the assembler always writes, populated when the constraint first appears rather than reconstructed later from the history.

The thresholds

The cut doesn’t happen when the window fills. It happens when a threshold is crossed, and the threshold exists because cutting at the limit means cutting in a hurry.

Up to around 70% occupancy there’s nothing worth doing: any saving costs more than it returns. Past that, the harness starts releasing what’s recoverable. Old tool output collapses into a line like “1,240 lines of output dropped, rerun pytest -q if needed”. The agent loses the content and keeps the way back.

Near 90% there’s nothing cheap left to drop, and compaction takes over: a model call that turns twenty turns into a summary of a few thousand tokens. It’s the most expensive operation in the set and the riskiest, because it’s the only one that erases information without leaving a route back.

Picking the threshold is a trade-off nobody documents. A low threshold compacts early and often, and the session loses detail that was still useful. A high one delays the cut until compaction has to swallow a lot at once, and summarising a lot loses more.

There’s a third case, and it’s the one that breaks in practice: the single jump that skips both thresholds. The agent sits at 62% and runs a command that returns 80K tokens. It goes from 62% to 102% without passing any decision point, and the harness now has a window that no longer fits. The good outcomes here are truncating the output before counting it, or writing it to a file and returning the path. The bad one, and the common one, is an emergency compaction with the giant output still inside what gets summarised.

The advertised window isn’t the usable one

The commercial number and the operational number deserve to be kept apart.

A 2024 benchmark evaluated 17 long-context models on 13 synthetic tasks, going past needle-in-a-haystack to include multi-hop tracing and aggregation. Nearly all showed large performance drops as length grows, and among those claiming support for 32K tokens or more, about half sustained acceptable performance at 32K¹. The test is synthetic and it isn’t your task, but the engineering implication is direct: budgeting against the advertised number means budgeting against a ceiling the model doesn’t use in full.

Position counts too. A 2023 study varied where the relevant information sat inside a long context and found higher performance at the beginning and the end, with a dip in the middle². In an agent, the thing choosing the position of each tool result is the harness. The practical rule that survives is a modest one: what must be obeyed goes at the edges, and the middle is where nice-to-have material lives.

The four ways to free space

Drop it. Delete the content and leave a marker that it existed. Zero cost, reversible whenever a path back exists.

Compress it. Rewrite the same content in fewer tokens without going through semantic summarisation. A 2023 method did token-level compression with a budget controller and reported up to 20x compression with little performance loss across four datasets, including GSM8K and ShareGPT³. That’s a result about task prompts, not agent history, and the transfer to the second case isn’t demonstrated.

Summarise it. Trade twenty turns for a paragraph. That’s what compaction does, and it costs a model call plus whatever the summary decided wasn’t important.

Move it out. Write to a file, a database or a subagent’s notes, and bring back only the relevant piece. The canonical formulation treats the window as main memory and everything else as secondary memory, with the model itself moving data between tiers through function calls⁴. It’s the only one of the four that increases the total information available instead of reducing it.

Dropping the oldest first looks like the obvious policy and isn’t a neutral one. A 2023 paper on attention caching found that the very first tokens in a sequence attract disproportionate attention regardless of content, and that keeping their state recovers much of what a naive sliding window loses⁵. The mechanism is internal to the model and doesn’t describe what the harness does with history, so it doesn’t transfer directly. What it does suggest is that “evict the oldest” is a decision with a cost, not a free operation.

Where this breaks

The loss is silent. The harness doesn’t announce a discard. The agent doesn’t know it forgot. The symptom arrives as a repeated question or a reversed decision, three iterations after the cut.

Counting tokens is expensive enough to be done badly. Tokenising the whole context every iteration costs time, so plenty of harnesses estimate by character count. The estimate runs high on code and low on accented text, and the error shows up as an overflow that shouldn’t have happened.

Compaction doesn’t compose. Summarising a summary loses again, and the second loss isn’t proportional to the first: whatever survived the first pass was already the most generic material there.

On-demand loading has latency. Fetching content only when needed trades tokens for round trips to disk or network. Over a 40-iteration task, five extra searches per iteration turn into a noticeable difference in wall-clock time.

Prompt caching fights aggressive pruning. Every time you rewrite the early part of the prompt to save room, you invalidate the cache and pay full price on the next call. Saving 8K tokens of window can cost more than keeping them.

The window can fill with content that isn’t yours. Tool output is externally sourced text entering the same sequence as the instructions. A web page, an issue, a dependency’s README: all of it consumes budget and simultaneously lands where the model looks for instruction. Managing context and managing trust are one decision at two points in the code, and treating them separately is how fetched content ends up carrying the same weight as what you wrote.

What to do

1. Measure the fixed parts today. System, catalogue and project config, in tokens. That’s the number you’ll pay on every call for the rest of the project.
2. Reserve the answer’s headroom before anything else. Overflowing at iteration 25 wastes everything spent up to there.
3. Truncate at the source, not at assembly. Every tool that can return a lot needs its own limit and a way to ask for more.
4. Mark what never leaves. Constraints and decisions need a structured place outside ordinary history. If they live in the middle of the conversation, they leave with the conversation.
5. Prefer dropping with an address over summarising. “File payments.py, lines 40 to 90, re-read if needed” beats a paragraph about what the file did.
6. Log occupancy per iteration. One log line with the five numbers from Figure 1 shows you within minutes where the window is going.

Step 6 is what turns this discussion into engineering. Without it, every context decision is a guess about a number nobody looked at.

Footnotes

1. Hsieh et al. (2024) built a synthetic benchmark with configurable length and task complexity, evaluated 17 long-context models on 13 tasks, and found sharp performance drops as length grows despite near-perfect accuracy on the simple retrieval test. ↩

2. Liu et al. (2023) varied the position of the relevant information inside a long context and measured higher performance at the edges with a dip in the middle, on multi-document questions and key-value retrieval. ↩

3. Jiang et al. (2023) performed token-level prompt compression with a budget controller and reported up to 20x compression with little performance loss across four datasets. ↩

4. Packer et al. (2023) proposed treating the context window as main memory and external storage as secondary memory, with the model moving data between tiers through function calls. ↩

5. Xiao et al. (2023) observed that the first tokens of a sequence concentrate attention regardless of content, and that preserving their state recovers much of the performance a sliding window loses. ↩