What is an AI agent?

An AI agent is a language model placed inside a loop, with access to tools and a stopping rule. It decides on an action, something executes it, the result returns to the context, and it decides again — until it finishes or gives up. What separates this from an ordinary call isn’t the model, it’s who decides what happens after the first response. Writing for this shape takes an agentic prompt, which is a different thing from an ordinary prompt engineering prompt.

The loop closing is the whole definition. Without it you have a call with tools, which is useful and is something else. A system that queries a weather API and formats the answer uses a tool and isn’t an agent: you decided to call that API, in code, before the model came anywhere near it. It becomes an agent when the decision to call again starts being made on the basis of what came back.

The model doesn’t execute anything

This is the most common misunderstanding, and it’s worth clearing up before anything else.

The model returns a structured request: “I want to use the fetch_order tool with argument id=8842”. It fetches nothing, opens no connection, touches no database. Your code executes, checking the argument against the declared schema, running the query and returning the result as a new message in the context, tagged as a tool result rather than as user text.

What reaches your code doesn’t always make sense. It can ask for a tool that no longer exists, because the catalogue changed mid-session, or send id="eight thousand". The text you hand back in those cases decides whether it recovers or spins. Query failed closes the road. id must be an integer, got "eight thousand" opens it.

That separation is what makes agents viable in production. Between the model’s decision and the action in the world there’s a point where you apply permissions, validate arguments, scope access and, when warranted, ask a human. That’s where you decide fetch_order runs straight through and cancel_order stops and asks. An agent without that point is a demo, not a system.

The rule has one exception. Some APIs run certain tools on their own side, web search and sandboxed code execution among them, and return the finished result without it ever passing through your process. The control point still exists, it just belongs to the provider and follows the provider’s policy. Worth knowing which of your tools work that way before you write your permission policy.

What changes versus a single call

The first consequence of that figure is cost, and it’s the counterintuitive one. The whole context is resent on every iteration, so a twenty-step task doesn’t cost twenty calls: it costs the sum of contexts that grow at each step. The curve is superlinear. The numbers are further down, with the assumptions on the table.

The second is latency. Twenty iterations are twenty waits in series, each one adding the model’s time to the tool’s. A task that would take three seconds in one call takes one to three minutes in an agent, and that changes the product, not just the infrastructure: you can’t hang an agent behind a button someone presses expecting an immediate answer.

The third barely shows up in comparisons. A single call at temperature zero is close to reproducible; an agent isn’t. Tool results change between runs, and a one-line difference in the output of step 3 leads to a different trajectory at step 4. Getting two distinct paths for the same request is expected behaviour, not a defect, and it dismantles most of the testing strategy that works on ordinary software.

The fourth is arithmetic. If each step has a 95% chance of coming out right and errors don’t self-correct, twenty steps in series finish with a 36% chance that everything held. At 99% per step it rises to 82%. That’s why cutting a task down to fifteen steps instead of thirty usually buys more than switching models.

None of the four is prohibitive when the alternative is human work. None of them is justified when a deterministic workflow would have done the job.

When it’s worth it

The useful question is: can you write the sequence of steps in advance?

If you can, use a workflow. It’s cheaper, faster, deterministic and debuggable. Most cases that arrive asking for “an agent” belong here.

If you can’t, because the next step depends on the previous result in a way you can’t enumerate, then an agent makes sense. Investigating a bug, researching until an answer turns up, navigating a structure you don’t know in advance.

Two requests that arrive sounding alike and aren’t. Invoicing a paid order has a fixed sequence: read the order, check the tax ID, work out the VAT, call the tax service, store the document, email the customer. Every step has a known error condition, and a model choosing the order only adds latency and a chance it skips the tax step. Working out why yesterday’s reconciliation closed £340 short has no sequence at all: where you look next depends on what turned up in the last place you looked.

A quick test: if you can draw the complete flowchart, you don’t need an agent. The test misleads in one case, and it’s a common one: some tasks have a flow that fits in five boxes and all their variation inside one of them. Invoice extraction is the example. The flow is always receive, extract, validate, store, and what you can’t enumerate is the forty layouts that arrive. The answer there isn’t an agent, it’s a workflow with one agentic step inside it. You keep the four boxes deterministic and pay for the loop only where it’s needed.

The six parts

Model, tools and a loop with a stopping rule are what make it an agent. Without all three there is no agent, and the third is the easiest to build halfway. Stopping needs more than one criterion, and the model announcing it’s finished is the least reliable of them. The others are a step ceiling, a token ceiling, wall-clock time and detected repetition: when the last three actions were identical, it isn’t making progress, it’s insisting.

Memory, permissions and harness are what separate a demo from a system.

Memory exists because the model has none. Between one call and the next it keeps nothing, and everything that looks like recollection is text your code resent. What survives the end of a session, and in what form, is your choice and nobody else’s.

Permissions decide how much damage is possible when it gets things wrong, and the useful question isn’t whether it will. It’s what happens when it does at three in the morning with nobody watching. An agent with read credentials on a replica gets it wrong and you read the log. The same agent with write credentials on the production database gets it wrong and you restore a backup.

And the harness, everything running around the model, usually explains more of the difference between two tools than the model choice does. How much of an output returns to the context, in what position it returns, what happens when the window fills: the model answers none of those, and all three change the trajectory.

The pattern that became the baseline

The structure most agents follow came out of a 2022 paper that proposed interleaving reasoning and action rather than separating them¹: the model writes what it’s thinking, picks an action, reads the result and repeats. It looks obvious now and wasn’t. Until then, reasoning and acting were treated as distinct stages, and joining them paid well: on two interactive decision-making benchmarks, ALFWorld and WebShop, the pattern beat imitation and reinforcement learning methods by 34 and 10 absolute points of success rate, prompted with only one or two in-context examples¹.

Two follow-ups still matter. One added reflection on its own failure: after getting something wrong, the agent writes in plain language what went wrong, and that text enters the context of the next attempt². The detail that matters is where the learning sits. Not in the weights, which don’t change, but in a text buffer your code stores, resends and you can edit.

Another showed tool use could be learned during training instead of described in the prompt³. The model annotated its own corpus, deciding where an API call would help, from a handful of demonstrations. That’s the line that led to models which now arrive already knowing how to emit a tool request without anyone explaining the format.

Read all three with the date in view. The scores have aged; the mechanisms haven’t.

Why it works in the demo and fails in production

This is the most common report from anyone putting a first agent live. The default reaction is to switch models, and it rarely helps, because the three most common causes aren’t the model.

Context. The demo has a four-iteration task and a clean environment. Production has accumulated history, and the window fills. When it fills, something is dropped or summarised, and what goes first is what sat at the beginning. The instruction to never touch an order with status invoiced was said once, at iteration 1, and doesn’t exist at iteration 18. It doesn’t announce that it forgot, it acts as though it never read it. A constraint that has to hold to the end is either reinjected or enforced inside the tool.

Permissions. In the demo you’re watching, approving every step and running with your own credentials. In production it runs alone under a service account, and what used to be a question on screen became an action taken. The question stops being “can it do this” and becomes “what happens when it does this to the wrong customer’s order”.

Tools. In the demo every call returns 200. In production the API returns 429, or an empty list that isn’t an error at all, or a 500 with an HTML body. If the message coming back is query failed, it repeats the same call, and repeats it again. If it’s 429, retry in 30 seconds, limit is 100 calls per minute, it waits. A tool’s error message is text written for a model to read, with every effect a prompt has, and it’s almost always written as though it were going into a log file.

Before switching models, read the trajectory: what it received after each tool, not what the terminal displayed. Most failures attributed to the model show up there as truncated context or output that said nothing.

What a twenty-step task costs

Order-of-magnitude numbers with the assumptions in the open, because the shape of the bill matters more than the values. Say system instructions of 2,000 tokens, a catalogue of five tools at 1,200, and each tool result coming back at 800. Those 3,200 fixed tokens go into every call, and each iteration adds the model’s request plus the result that came back to the history, somewhere near 900 tokens.

The first call costs 3,300 input tokens. The twentieth costs 20,400. The twenty together come to over 237,000, against 3,300 for the single call. Twenty steps cost roughly seventy times one call, and it’s that ratio, not the price per token, that explains the surprise on the invoice.

Prompt caching cuts a good deal of it, and in an agent it cuts well, because each iteration starts with exactly what the last one had: the prefix repeats almost whole. Tokens read from cache cost a fraction of the input price. What caching doesn’t change is the shape of the curve. It lowers the floor and leaves the growth where it was.

Two decisions move that number more than anything else. How many tools stay loaded, since the catalogue is resent on every call: forty cost ten times what five cost, and the thirty-five nobody will use cost the same as the rest. And how much of each output returns: handing back a grep with 3,000 lines or handing back the 20 that matter is the difference between filling the window at iteration 8 and reaching 25.

Count your own rather than trusting these. It’s one line of code inside your own loop.

Where to start

1. Try it without an agent first. Single call, then a prompt chain, then a workflow. Only climb a rung when the one below fails on a real case, not an imagined one.
2. Start with two or three tools. A large catalogue confuses the model and eats context on every call. A new tool goes in when a trajectory shows it was missing, not because it looked useful on paper.
3. Put a step limit in from day one. Twenty is a reasonable start and you tune it with data. An agent with no ceiling becomes an invoice.
4. Decide what it can never do alone before deciding what it can. The forbidden list is short and you write it in ten minutes; the permitted list is never finished.
5. Log the whole trajectory. Every request sent and every result as the model received it, not the final answer. When it fails you’ll need to see what it saw.

Item 5 is the one almost nobody does at the start and everybody wishes they had at the first incident.

Footnotes

1. Yao et al. (2022) introduced the interleaved thought-and-action pattern that became the baseline structure for most agents, and measured 34 and 10 absolute points of success rate over imitation and reinforcement learning methods on ALFWorld and WebShop. ↩ ↩²

2. Shinn et al. (2023) added a step of verbal reflection over the failure, stored in an episodic memory buffer for the next attempt, with no weight update. ↩

3. Schick et al. (2023) showed tool use can be learned self-supervised during training, from a handful of demonstrations per API, rather than described in the prompt. ↩