How does an AI agent's loop work?

An AI agent loop has four stages per turn. The whole context goes to the model; the model returns reasoning plus a tool request; your code validates and executes that request; the result enters the context as a new message. Then the next turn begins, with a larger context. Only the second stage runs on the model.

That asymmetry matters most and shows up in diagrams least. Three of the four stages are your code, and in them you decide what the model gets to read next. The shape of interleaving reasoning and action in one sequence came from the ReAct pattern, published in 2022, and has barely changed since¹.

The two consequences that surface first in production follow from that. The context grows every turn and is resent in full, which is what makes cost climb superlinearly. And because everything shares one window, a long task eventually needs some form of isolation to keep apart what shouldn’t be mixed.

One turn, with the real contents

It helps to follow a concrete turn, inside an agent investigating why a batch of recurring charges failed overnight.

Stage 1: the context leaves here. On the first turn it has four parts: the system instructions, the descriptions of the available tools, the user’s request, and nothing else. Somewhere near 3,100 tokens. It matters that this is a list of messages, not a string: each item carries a role, and the role changes how the model treats the content.

Stage 2: the model answers with two kinds of block. One is text, the reasoning. The other is structured: the tool name, the arguments, and an identifier for that particular call. The identifier exists because each response has to be matched back to the right request when several calls go out at once.

Stage 3: your code executes. Everything that isn’t the model happens here. You check that the tool exists, validate arguments against the schema, apply that user’s permissions, run the operation, and catch the error. A whole turn can die in this stage, with the model never called again, if validation refuses.

Stage 4: the result comes back as a tool message. Not as user text. The distinction is real and it has an effect: models treat a user message as an instruction and a tool result as an observation. Returning an API error in the user role is a quiet way of telling the model the user asked for it.

The ordering of these messages is strict. Each result has to appear directly after the request that produced it, matched by identifier, and most APIs reject a history that is missing the pair. This matters a lot when you truncate or summarise: dropping a tool request without dropping its result leaves the history invalid, and the error that comes back is about format rather than content, which usually costs half an hour to diagnose.

And then turn 2 starts with a context that now holds 3,800 tokens, because the tool request and the result stayed in the history. Nothing was removed.

Perceive, decide, act is a borrowed name

The perceive, decide, act cycle comes from robotics and from the agent literature that predates LLMs, where there was an actual sensor. The name stuck because the shape really is the same, and it misleads on one point.

In an LLM agent there is no perception. What the model “perceives” is the text your code wrote into the context, in the order your code chose to write it. If read_log returns 3,000 lines and you hand over all 3,000, that is the perception. If you hand over the 20 lines around the error, that is the perception. There is no world behind it the model can consult on its own.

That moves where the work happens. Most of the effort in improving an agent is not in the prompt or the model: it is in deciding what each tool returns, in what shape and at what level of detail. That is stage 4 of the loop, and it is the cheapest one to change.

It also explains a failure that reads as stubbornness. An agent that keeps querying the same table is often being told nothing by the table. An empty result serialised as [] and an empty result serialised as no rows matched status 'pending' between 02:00 and 04:00 are the same fact and produce very different next turns.

The context grows every turn

With 3,100 fixed tokens and roughly 1,100 added per turn across request and result, turn 18 lands near 23,000 input tokens. That number isn’t alarming by itself. The problem is the shape of the growth and what it does to the model’s attention.

A 2023 paper measured how performance shifts with the position of the relevant information inside the context. The result was a U: models do better when what matters sits at the beginning or the end, and drop significantly when it sits in the middle, including in models marketed for long context². In an agent loop, the beginning is the original instruction and the end is the latest tool result. The middle is the entire history of the investigation.

Two things follow, both practical. A constraint that must hold until the last turn cannot live only in the first message: either it gets reinjected near the end, or it becomes validation inside the tool. And the result that matters most should be the last thing to enter, which in most loops happens for free.

Be sceptical of the obvious fix, which is summarising the history when it gets large. A summary is an extra call, it costs tokens, and what it drops is decided by a model that doesn’t know which detail you’ll need on turn 22. It works, but it’s a trade, not a repair.

How many turns are normal

The useful range runs from three to forty, with most tasks that work well landing under fifteen.

Below three, the loop isn’t earning its keep: if the task resolves in two calls in a known order, a workflow delivers the same thing cheaper and faster. Above forty, the pattern is almost never slow progress. It is repetition: the same three actions alternating, or the same query with a slightly different argument four times running.

There’s simple arithmetic behind that. If each turn has a 95% chance of coming out right and errors don’t self-correct, fifteen turns in series end with a 46% chance that everything is correct. Thirty turns drop to 21%. Shortening the task usually pays more than raising the cap.

How long one turn takes

Order-of-magnitude numbers with assumptions on the table. A model call with 8,000 input tokens and 300 output tokens takes between 3 and 8 seconds, and the time grows more with output size than with input size. The tool takes what it takes: an indexed query answers in 50 ms, a scan across a full day of logs takes a few seconds, a call to an external service with a retry can pass twenty.

Fifteen turns at 5 seconds of model plus 2 of tool land close to two minutes. None of that parallelises along the main axis, because each turn depends on the previous result. What does parallelise is inside a turn, when the model asks for three independent reads at once, and that gain is real but limited to the part with no dependency.

Two things move this number more than the choice of model: how many turns the task needs, which you control by cutting scope, and how long the slowest tool takes, which is usually a missing index somewhere.

The four stopping criteria

Step cap. The simplest and the most effective. Twenty is a reasonable start and you tune it with data, by looking at the turn distribution of successful runs.

Token cap. Necessary because steps don’t have a fixed size. A single tool that returns a large file can burn in one turn what ten normal turns would.

Wall-clock cap. This is what protects the user and the queue. A task past ten minutes rarely ends well, and while it runs it holds resources.

Repetition detected. The cheapest to implement and the least used: keep the tool name and arguments of the last three calls; if all three are identical, the agent isn’t progressing, it’s insisting. Stopping there saves most of the waste.

The fifth signal is the model declaring it is finished, and it is the only one that can be wrong with nobody noticing. A 2023 paper examined intrinsic self-correction, with no external feedback, and found that reasoning performance doesn’t improve after revision and in some setups degrades, because the model has no way to know it was wrong³. If it can’t reliably judge its own answer, it can’t reliably judge whether the task is done either.

The answer isn’t to drop that signal, it’s to pair it with verification. Reflecting on a failure, written in natural language and kept for the next attempt, improves results when some external signal says the previous attempt failed⁴. What changes the picture is where the signal comes from: a test that runs, a comparison against expected state, a schema validation. Not the model’s opinion of itself.

Where the loop breaks

No criterion exists. This is the amber box in the figure. “Keep going until it’s fixed”, with no cap, facing a permanent error, runs until somebody checks the bill.

The window fills and the beginning disappears. When the history passes the limit, something gets dropped, and what sat at the start goes first. The instruction given on turn 1 no longer exists on turn 18, and the model doesn’t announce that it forgot.

The error message says nothing. Query failed makes the model repeat the same call. Argument date must be ISO 8601, received "yesterday" makes it correct. Tool error text is a prompt, with all the effects of a prompt, and it is nearly always written as though it were going into a log file.

The tool returns too much. A 3,000-line response fills the window in six turns. Trimming at the tool is almost always the highest-return edit in an agent that loses its way.

Parallelism hides a dependency. Firing three calls together cuts latency when they’re independent. When the second needed the first one’s result, the model gets two observations that don’t fit together and the next turn starts from an inconsistent state.

How to instrument it

1. Log the whole trajectory, every request sent and every result exactly as the model received it. Not the final answer, not what the terminal displayed.
2. Count tokens per turn and keep the series. It’s one line of code and it’s what shows which tool is filling the window.
3. Measure the turn distribution of runs that succeeded. The cap comes out of that number, not out of a guess.
4. Detect repetition from day one. Three identical actions in a row is a five-line stopping condition.
5. Return tool errors written to be read. What happened, what was expected, and what to do next, in one sentence.
6. Run the same task eight times and compare trajectories. Two runs that diverge at turn 4 tell you exactly which observation is being read two ways.

Item 1 is what almost nobody has at the start and the first thing everybody reaches for during the first incident.

Footnotes

1. Yao et al. (2022) proposed interleaving reasoning and action in one generation sequence, with the action’s result returning as an observation, which is the loop structure still in use. ↩

2. Liu et al. (2023) measured a U-shaped curve: performance is highest when the relevant information sits at the start or the end of the context and drops significantly when it sits in the middle. ↩

3. Huang et al. (2023) found that, without external feedback, revising its own answer does not improve reasoning and in some setups makes the result worse. ↩

4. Shinn et al. (2023) showed gains from turning a failure into reflection text kept for the next attempt, in a setup that depends on an external signal saying the previous attempt failed. ↩