What is the difference between structured outputs and function calling?

Both run on the same machinery: a schema that constrains decoding. The difference is what happens next. With structured outputs the response is the object and the call is over. With function calling the model asks your code to run something and waits for the result. If nothing is going to run, you want the first one.

The confusion is fair, because the two sit in the same corner of every provider’s docs and cure the same immediate symptom, which is getting valid JSON out of the model. One of them solves shape, though, and the other solves control.

This article is about the choice. Declaring the shape belongs to structured outputs, and the execution loop, with its tool registry, error handling and round limit, belongs to function calling explained.

The machinery is the same

Start with where the difference is not. Both take a JSON Schema, and both compile that schema into the same constraint over sampling: an automaton tracks what has been emitted and zeroes the probability of any token that would lead to an invalid sequence¹. Under strict mode the syntax guarantee is identical on both paths, because it is literally the same code running.

That disposes of the question people ask most: which one is more reliable at producing valid JSON? Neither. When a provider does show a measured difference, it comes from strict mode being implemented on one endpoint and not the other, or from the two endpoints covering different slices of the specification. That is a product difference, not a methodological one, and it moves between releases.

What genuinely changes is the envelope. In one case the object arrives as the content of the response and the call is finished. In the other it arrives inside a tool-use block carrying a name and an id, and the model has stopped mid-turn waiting for you to send back a result tagged with that id.

The difference is what comes after

Structured outputs means one inference and one code path. You call, receive the object, validate it and move on. The schema applies to every response and the model has no choice to make about it.

Function calling means at least two inferences and a loop. The model decides whether to call a tool, which one, and with what arguments; your code runs it; the result comes back as a new message; and the model runs again, now with that result in context. It can request another tool, request the same one with different arguments, or stop and write the final answer.

That word, decides, is the whole difference. It defined the mechanism from the start: the work described in Toolformer is a model learning which APIs to call, when to call them, what arguments to pass, and how to fold the result into what comes next². None of that is about format. It is about a decision the model makes, and one that can go wrong in four distinct ways.

Why “a tool just to format” became a habit

The pattern has a plain historical explanation. Function calling reached commercial APIs before any response-level schema existed. Anyone who needed a guaranteed shape in 2023 declared a fake tool, forced the call and read the arguments as if they were the answer. It worked, and it stuck.

In 2026 that is debt in most codebases. The tell that you are paying it is tool_choice pinned to one specific tool: forcing the tool removes exactly the choice that justified using tools at all.

Two exceptions stay legitimate. One is the provider that only offers strict schemas on tool parameters and not on the response, in which case the tool is the only route to the guarantee. The other is when you want the model to choose between different answer shapes — one tool per shape is a natural discriminated union, and the chosen tool name is already the discriminator.

What changes in your code

The complexity bill is lopsided, and it usually gets underestimated at decision time.

On the structured outputs side you write one path: call, extract, validate, use. The one failure is the object not validating, and the treatment is a retry with the error message attached.

On the function calling side you write a dispatcher mapping tool name to function, argument validation at your own boundary (the provider’s schema guarantees the shape, not that the id inside it exists), the execution itself, a translation from execution errors into something the model can read, a round limit so the loop cannot spin forever, and a policy for when the model asks twice for an action that is not idempotent.

There is one upside that rarely gets mentioned: the loop leaves a trail. Every decision becomes a record with a name, arguments and a result, which is debugging material that structured outputs never produces. If your problem is understanding why the system did what it did, the loop earns its keep even where the format did not require it.

What streaming changes

Both stream token by token, and what you can do with the partial flow differs.

With structured outputs you watch an object assemble left to right. You can render field by field as they land, provided your parser tolerates incomplete JSON and you accept that a value may be rewritten before it closes. That is what powers an interface filling a form while the model is still answering. Note that field order in the schema becomes the order things appear on screen, which stops it from being a purely technical decision.

With function calling the tool name arrives before the arguments, because it is the first thing the model emits in that block. That lets you show “checking stock” at the instant the decision was made, with the arguments still arriving. It is very little code and it covers most of the perceived slowness in a multi-round loop, where time to the final answer is the sum of every inference.

Which one for which job

The rule fits in one question: does something get executed, and does the result need to go back to the model?

No, nothing runs. Structured outputs. Entity extraction, classification, scoring, summarising into fixed fields, translation with metadata. In all of these the model produces a final piece of data, and that data is the answer.

Yes, and the model needs the result. Function calling. Checking stock, searching an index, reading a file, opening a ticket whose number goes into the reply. The common thread is that the model cannot finish without something only your system knows.

The middle case, which almost everyone gets wrong. The model picks one action out of several, your code performs it, and the conversation ends. It looks like function calling because there is an action, and it is not: nothing goes back to the model.

The stock example is ticket routing. A support message arrives, the model decides whether it belongs to billing, technical or cancellation, and your code moves the ticket. Built as tools, that becomes three definitions in the prompt, a decision about which one to call, and a second inference for the model to announce it is done. Built as a structured output, it is a destination field with three enum values and a justification field ahead of it. One inference, no loop, same guarantee on the value. You halve the calls, and the possibility of the model answering in prose instead of choosing disappears with them.

Where each one fails

Function calling fails on the choice, not the format. That failure surface simply does not exist for structured outputs. Gorilla flagged the pattern in 2023: the hard part of API calls was never producing JSON, it was producing correct arguments and not hallucinating how the API is used, and even the strongest models of the day stumbled there; the authors showed that pulling documentation in at call time substantially reduced the hallucination³.

The choice has three parts, and they fail separately. API-Bank split planning, retrieving and calling apart in a runnable system with 73 tools and 314 annotated dialogues covering 753 calls, precisely because a single success rate hides which of the three broke⁴. If your loop misbehaves, measuring the three together will not tell you where to intervene.

Inconsistency hurts more than the average. τ-bench evaluated function-calling agents in conversations with a simulated user under domain rules and found under 50% task success even for the strongest models of the time. The more uncomfortable number was a different one: running the same task eight times, consistent success fell below 25% in the retail domain⁵. For a flow that performs real actions, being right on average matters far less than being right every time.

Structured outputs never abstains. The schema has required fields, so the model fills the required fields, whether or not the information exists in the input. It is the most common failure in extraction and the quietest. The fix is to declare fields nullable and write down what to do when nothing is found, rather than expecting the model to leave a blank on its own.

Function calling also fails by overreach. The mirror image of abstaining is calling when there was no need: the model checks stock to answer a question that does not depend on stock, and you pay two inferences for an answer the first one already had. An over-promising tool description is the usual cause, and the fix is to write into the description when not to use the tool, not only what it does. It is worth measuring the call rate on inputs that should trigger no tool at all; it tends to run higher than anyone expects.

Tool definitions occupy the prompt. Each tool with a decent description costs roughly 100 to 200 tokens, on every call, and the whole list competes with the real input for the model’s attention. Twenty tools are several thousand fixed tokens and a harder decision to get right.

Both write in schema order. If the result field precedes the justification field, the justification is a rationalisation written afterwards. That holds for the response schema and for the tool parameter schema alike.

Cost

Structured outputs costs one inference. The premium over a plain call is the schema compilation the first time and nothing meaningful after that.

Function calling costs one inference plus one per tool executed, and every round resends the whole conversation, which grows with the results you fed back. A three-tool loop over a 4,000-token prompt is four inferences with the input climbing at each step. Prompt caching covers the stable part, which is the system message plus the tool definitions, and that is why the stable part has to sit ahead of anything that varies.

Where to start

1. Answer the loop question before picking an API. Does something execute and does the result return to the model? That alone decides it.
2. If you are forcing one specific tool, switch to structured output. Same result, fewer moving parts.
3. If the model picks and your code runs it without reporting back, use an enum. One less inference per request.
4. Measure planning, retrieval and calling separately if the loop is real. The aggregate number does not point anywhere.
5. Run the same task several times before shipping. In a flow that performs actions, the variation between attempts matters more than the mean.

Footnotes

1. Willard and Louf (2023) describe guided generation as transitions between finite-automaton states over a pre-computed vocabulary index — the same machinery behind both response schemas and tool parameter schemas. ↩

2. Schick et al. (2023) trained a model to decide which APIs to call, when to call them, what arguments to pass and how to incorporate the result into the continuation, in a self-supervised way. ↩

3. Patil et al. (2023) identified that the difficulty in API calls lies in producing correct arguments and avoiding hallucinated usage, and showed that coupling document retrieval at call time reduces hallucination and keeps up with version changes. ↩

4. Li et al. (2023) built a runnable system with 73 tools and annotated 314 dialogues covering 753 calls, evaluating planning, retrieval and calling as separate abilities. ↩

5. Yao et al. (2024) evaluated function-calling agents against a simulated user under domain policies: under 50% task success, and across eight repetitions of the same task, consistent success below 25% in the retail domain. ↩