What is an agent harness and why does it matter so much?

An agent harness is all the software sitting between the model and the world in an AI agent: what enters the context window, which tools exist and how they’re described, what needs confirmation, where the command runs, and when to stop spending. The model picks the action. The harness decides what it can see, what it can do, and what that costs.

It matters because two tools running the same model, on the same day, with the same request, produce results different enough to look like different models. Almost every public argument is about which model to use. Much of the gap lives in the layers wrapped around it.

That makes diagnosis hard. When an agent fails, the first instinct is to swap models, and the culprit is usually outside. This article is about what that outer layer does, not about adopting a library versus writing the loop yourself, which is a neighbouring and separate decision.

What happens between one turn and the next

The model does exactly one thing: it takes a sequence of tokens and returns another. Everything that looks like agent initiative happens outside it.

One full iteration, in the order it runs:

1. The harness assembles the request: system instructions, conversation history, results from earlier tool calls, the catalogue of available tools, project configuration.
2. The model answers with text and, possibly, a tool request.
3. The harness parses that request and validates the arguments against a schema.
4. It checks whether this particular action needs human confirmation.
5. It executes, inside some filesystem, network and time limit.
6. It gets the output, which might be 200 bytes or 40 MB, and decides how much of it comes back.
7. It reinjects the result into the context, in some format and some position.
8. It updates the persisted session state.
9. It checks the token and step budget.
10. It decides whether to call the model again.

The model takes part in step 2. The other nine are somebody’s code, making choices nobody documented because they look like implementation detail.

None of those steps is neutral. Step 6 alone carries a decision that reshapes the whole task: when a search returns 3,000 lines, the harness can keep the first 50, keep the last 50, summarise them with a second model call, or write them to a file and hand back the path. All four are defensible. They produce trajectories that look nothing alike.

The harness also decides what the model never sees. A command that exits with an error can come back as the full stderr, as a one-line summary, or as nothing beyond “failed”. In the third case the agent retries blind, because the information that would have solved it was thrown away before reaching it. Nobody writes that decision down, and it accounts for a whole class of pointless repetition.

Why the public conversation is almost entirely about the model

A model has a name, a version, a cutoff date, benchmark scores and a launch page. A harness has a release number and a changelog nobody reads.

The effect is that the one visible variable in the conversation is the one that varies least between competing tools. As of July 2026, most coding agent tools run the same three or four frontier models, often the same one by default. The experience gap between them stays wide. It can’t be coming from the model, because the model is the same.

There’s a second, less comfortable reason: a harness is hard to measure in public. A leaderboard reports a model and a score, and the scaffold becomes a footnote when it appears at all. Two submissions with the same model and distant scores get read as experimental noise rather than as the headline result.

What has been measured with the model held fixed

Three published results isolate one harness decision and show the size of the effect without swapping models.

The action format. A 2024 study compared three ways for an agent to request an action: JSON, text in a fixed format, and executable Python code. They tested 17 models on API-Bank and on a benchmark of their own. The code version came out up to 20% higher in success rate¹. Nothing about the model changed. What changed was how the harness represents an action and what it accepts back.

The interface between agent and machine. Another 2024 paper started from a plain observation: terminals and editors were designed for humans, with scrolling, colour and paging that don’t help a reader who takes everything in at once. The authors built an interface specifically for the agent — file navigation, editing with verification, test execution — and reported 12.5% pass@1 on SWE-bench, the state of the art when the work came out². The number has aged: by July 2026 it reads as a historical marker rather than a reference point. What hasn’t aged is the finding, which was the paper’s declared goal: interface design changes agent behaviour.

Where information sits in the context. A 2023 study measured what happens when the relevant information moves around inside a long context. Performance is highest when it’s near the beginning or the end and drops in the middle³. The experiments used models of that era, and the magnitude has changed since. The part that still holds is weaker and more useful: position isn’t neutral, and the thing choosing the position of every tool result is the harness, not you.

None of the three is a result about intelligence. They’re results about plumbing, and plumbing is what most teams never instrument.

The six areas where the decisions live

The seven layers from the previous figure collapse into six decision areas. The sandbox follows from the permission policy, and subagents are a form of context management: each one gets a clean window and hands back only a summary.

Context. What fraction of the history survives, what gets summarised, what gets dropped, what gets re-read from disk when it’s missed. It’s the layer with the most effect and the least visibility, because the loss is silent: the agent doesn’t announce that it forgot, it just behaves as if it had never seen.

Tools. Which ones exist, how they’re described, how many stay loaded at once, and how the harness offers 80 of them without spending context on the 78 that won’t be used. A tool description is a prompt, with every property of a prompt, and it’s almost never treated as one.

Extension. How you add capability without touching the harness code: MCP servers, scripts, hooks, stored procedures. It sets the ceiling on what you can do without switching tools.

Permission. What runs straight through, what stops and asks, what is never allowed. Erring permissive produces an incident; erring restrictive produces an agent that interrupts so often it’s cheaper to do the work by hand.

Session. What survives the end of a run, and what happens when the window fills mid-task. Picking up where it left off is a harness capability, not a model one — the model has no memory between calls.

Cost. Step limits, token limits, what triggers a warning, what halts. Without this layer you find the ceiling on the invoice.

The six contaminate each other, and that’s what makes diagnosis treacherous. A large tool catalogue burns context, which brings compaction forward, which makes the agent forget a decision from two iterations ago and redo the work, which burns budget. The symptom shows up in cost. The cause is in the tool layer. Swapping models moves neither.

Same model, two harnesses

Take the task “find out why the shipping calculation is ignoring the coupon”. The model is the same on both sides.

Harness A offers a read tool that returns whole files. The agent opens 40 of them, the window fills on iteration 12, compaction summarises the history, and the summary drops the exact line that held the clue. It restarts the search with less information than it had before, and that’s the expensive part: the second pass costs more tokens and arrives at less.

Harness B offers pattern search, reading by line range, and somewhere to write findings outside the context. The agent reads three excerpts, notes the line number, and what it noted is still there after compaction.

The difference isn’t reasoning. It’s which tools exist and what happens to their output.

Where the harness doesn’t help

More harness is not better, and this is the most important correction in this article.

A complex scaffold loses to a fixed pipeline more often than you’d think. A 2024 paper built a three-phase process — localise, repair, validate — without letting the model choose its next step, and reached 32.00% on SWE-bench Lite at $0.70 per issue, ahead of every open-source agent at the time⁴. The reasonable reading is that part of what gets called harness is complexity standing in for a capability the model already has, and getting in its way.

A harness can’t fix inconsistency. The same agent, on the same task, eight times running, does not produce the same result eight times. A 2024 benchmark measured this with a metric requiring success on all k attempts and reported pass^8 below 25% in the retail domain, with gpt-4o solving under 50% of the tasks in a single attempt⁵. No permission or context layer turns that into reliability; they only lower the cost of each failure.

A harness sticks to the model it was tuned against. Tool descriptions, expected output formats and system instructions get tuned against one model family. Switching families usually drops results, which makes a harness look better than it is on its house model and worse than it is off it.

Every layer costs. A permission check adds a round trip to the user. A sandbox adds startup latency. A tool catalogue adds tokens to every call. A layer that prevents a rare failure and costs 3% on every run may not pay for itself.

Attribution gets scrambled in practice. When a tool improves from one month to the next, it has usually changed its default model and touched three harness layers in the same window. From outside you can’t separate them. That’s why the useful comparison is the one you run yourself, with the model pinned, and not a close reading of a release note.

What the harness costs, to an order of magnitude

The measurable part is the tool catalogue, because it’s resent on every iteration. A tool schema with five parameters and short descriptions usually lands between 150 and 400 tokens, depending on the tokeniser. Forty loaded tools put 6,000 to 16,000 tokens into every call before any history. Over a 30-iteration task that becomes 180,000 to 480,000 tokens of catalogue alone.

Prompt caching removes most of that while the prefix stays stable, which turns “which tools stay loaded” into a cost decision that compounds. Treat the numbers above as an order of magnitude: count your own, it’s one line of code and it beats any estimate in this article.

How to measure the harness you already use

1. Log the full trajectory. Not the final answer: every request sent and every tool result as the model received it. Without that, failure analysis is guesswork.
2. Count tokens per part of the request before arguing about models. System instructions, tool catalogue, history, results. One of the four is usually eating more than you think.
3. Read tool output the way the model got it. The terminal shows colour and paging; the model gets text truncated at a point somebody chose.
4. Pin the model and change one layer at a time. Two variables at once produce no conclusion.
5. Run the same real task through two harnesses with the same model. Twenty tasks from your own repository say more about your choice than any public leaderboard.

All five carry a declared bias: they assume the answer sits in what the model received rather than in the model. That’s the cheapest place to look first, and the three measured results above are the reason to start there.

Footnotes

1. Wang et al. (2024) compared JSON, fixed-format text and executable Python as the action space across 17 models, measuring up to 20% higher success rate for the code version. ↩

2. Yang et al. (2024) built an interface designed for the agent rather than reusing the human one, and the declared goal of the work was to measure how much that interface design affects performance. ↩

3. Liu et al. (2023) measured a performance drop when the relevant information sits in the middle of a long context, on multi-document question answering and key-value retrieval. ↩

4. Xia et al. (2024) showed that a fixed three-phase process, with no autonomous next-step decision, beat the open-source agents of the time on SWE-bench Lite at a lower cost per instance. ↩

5. Yao et al. (2024) proposed the pass^k metric, which requires success on all k attempts, and measured a sharp reliability drop between one attempt and eight. ↩