How do you make an LLM return structured data?

There are three ways to make an LLM return structured data, in increasing order of guarantee: ask for the format in the prompt, validate the answer against a schema and retry when it fails, or constrain decoding so an invalid token is never sampled at all. Only the third guarantees syntactically valid JSON. The first two are cheaper and handle most cases.

Choosing between the three steps is almost always a cost decision. Forcing valid JSON covers the shortest path when all you need is a response the parser will accept; JSON Schema and Zod covers the validation half; and fault-tolerant parsing covers what to do when the answer arrives broken anyway.

Before choosing, it helps to know why the format breaks at all. The cause sits in how the model generates text, not in a badly written prompt — which is why a firmer instruction buys so little here, unlike most of prompt engineering.

Why the format breaks

The model picks one token at a time, sampling from a distribution over the whole vocabulary. Nothing in that process carries the fact that there’s an open brace waiting to be closed, or that the total field was declared as a number. JSON validity is a property of the entire sequence; sampling is local. The model gets the format right because it saw a lot of JSON in training, not because anything stops it from getting it wrong.

In practice the breakage shows up in four shapes, and they’re worth telling apart because each has a different fix.

A markdown fence around it. The model returns exactly the right JSON, wrapped in ```json. The most common failure and the cheapest to fix, because the data itself is intact.

A trailing comma. A , before the } or the ]. Perfectly readable to a human, rejected by JSON.parse and by json.loads.

Quotes. Two distinct variants: typographic quotes (“ ”) instead of straight ones, usually when the field text came from formatted content; and unescaped straight quotes inside a string, which close the value halfway through.

Truncation. The sneakiest one. The JSON is flawless right up to the point where the tokens ran out. The symptom is failing only on long cases, which looks like randomness until you check the response’s stop field (stop_reason or finish_reason, depending on the provider) and find max_tokens.

Here’s one response carrying all four at once, which is the realistic case:

Sure! Here's the JSON:

```json
{
  "customer": "Corner Bakery",
  "items": [
    {"sku": "PF-001", "qty": 12, "note": "flour "type 1""},
    {"sku": "PF-002", "qty": 3,},
  ],
  "total": 148.

A markdown fence with prose before it, unescaped quotes in note, trailing commas in two places, and truncation in the middle of total. Each one wants a different treatment, and none of them is “ask more firmly”.

Step 1: ask in the prompt

The cheapest option, and enough for a prototype. Three things genuinely help:

1. Show the exact output rather than describing it. One worked example conveys field names, types and date format in a single shot.
2. Say what must not appear. “Reply with the JSON object only, no code fence and no text before or after” removes most of the fences.
3. Leave headroom. Half of the truncation failures are a max_tokens sized for the average case rather than the worst one.

One classic trick died recently and is worth recording: prefilling the start of the assistant’s reply with { so the model had to continue from inside the object. As of July 2026, current Anthropic models, from the 4.6 generation onward, reject a request whose last message is from the assistant with a 400. Where prefill still exists it still works; where it doesn’t, native structured output is the replacement.

This step fails in a specific way: it works across your ten test cases and breaks in the first week of production, on an input with a quote in the customer name.

Step 2: validate afterwards

Here you accept that the output may come back wrong and treat that as a normal case rather than an exception. The pattern has three parts: extract, validate, retry.

Extract means slicing the object out of whatever came back. Cutting from the first { to the last } handles markdown fences and surrounding prose with no cleverness at all.

Validate means checking against a schema, not just against the parser. JSON.parse happily accepts {"qty": "twelve"}; your schema shouldn’t.

import { z } from 'zod';

const Order = z.object({
  customer: z.string().min(1),
  items: z.array(z.object({ sku: z.string(), qty: z.number().int().positive() })),
  total: z.number(),
});

function extract(raw: string) {
  const i = raw.indexOf('{');
  const j = raw.lastIndexOf('}');
  if (i < 0 || j < i) throw new Error('no object in the response');
  return JSON.parse(raw.slice(i, j + 1));
}

Retry is the part most people do badly. Re-sending the same prompt and hoping the model gets it right this time throws away the most useful thing you have: the error message. Hand the validator’s error back to the model along with its previous answer, and the second-attempt success rate climbs sharply.

const repair = [
  { role: 'assistant', content: raw },
  { role: 'user', content: `That failed validation: ${err.message}\n` +
      `Return the corrected object only.` },
];

A repair pass like this is cheap and measurable. In Tam and colleagues’ study, asking a small model to reformat outputs that had failed to parse improved the final scores in both JSON and YAML¹.

Step 3: constrain the decoding

The third step changes the nature of the problem. Instead of judging text after it exists, you intervene at sampling time: at each position, tokens that would lead to an invalid sequence get zero probability. The model isn’t persuaded to produce valid JSON — it becomes unable to produce anything else.

The mechanism is a mask over the logits. The schema, or the grammar, is compiled into an automaton; the automaton’s state after the tokens emitted so far determines which vocabulary tokens are acceptable next, and everything else is zeroed before sampling. After {"qty": only digits and a sign get through; after {"customer": only a quote does.

The obvious objection is cost. Testing the whole vocabulary at every token is expensive: even a small vocabulary like GPT-2’s has 50,257 entries, and naive implementations walk all of it at each step. Willard and Louf showed how to build an index from automaton state to allowed tokens ahead of generation, which brings the per-token work down to constant cost on average². That’s what moved the technique out of papers and into API surfaces.

The idea reaches past JSON. Context-free grammars describe the output space of a broad class of tasks (SQL, function calls, agent action sequences), and the same masking mechanism serves all of them³.

As of July 2026, providers expose this in two shapes. On the Anthropic API, output_config.format with a json_schema constrains the response, and strict: true on a tool definition constrains that tool’s parameters; the schema has to declare additionalProperties: false and list its required fields. A new schema pays a compilation on its first call and is cached for 24 hours. Two practical consequences: the first request for a schema is slower than the ones after it, and rebuilding the schema on every call throws that cache away.

Format is only the first front

Getting the format right is the visible part. The next three problems arrive in order, each costing more than the last: making sure the content makes sense, deciding what to do when it doesn’t, and holding that up under volume — batching, rate limits, streaming, retry idempotency.

Where this fails

Valid JSON is not a correct answer. Constrained decoding guarantees that {"category": "billing"} will parse and that billing is in the enum. It does not guarantee that the ticket was about billing. This is the most expensive confusion in the topic, because the system stops failing loudly and starts failing quietly.

Field order in the schema changes the result. Tam and colleagues found the clearest case of this: in 100% of GPT-3.5 Turbo’s JSON-mode responses, the answer key appeared before the reason key, which turned a chain-of-thought task into direct answering and dragged performance down¹. The model writes in the order the fields appear; if the result comes before the reasoning, there is no reasoning.

The loss doesn’t come from parsing. In the same work, LLaMA 3 8B had a 0.148% parse error rate on the Last Letter task in JSON and still showed a 38.15% performance gap against free-text output¹. Treat that as a warning against the easy explanation: if structuring hurt your numbers, the parser is probably not the culprit.

A badly implemented constraint really does hurt. Beurer-Kellner and colleagues measured a JSON-encoded version of GSM8K with Mistral 7B and five-shot prompting: 41.5% accuracy unconstrained, 30.8% with naive constraining, 41.8% with minimally invasive constraining⁴. The cause is subword misalignment — the mask forces a token split the model never saw in training, and the next distribution lands off-distribution. The practical reading is that library quality matters, not just the presence of a constraint.

JSON Schema coverage is partial. A 2025 benchmark evaluated six constrained decoding frameworks over 10K real-world schemas and found meaningful coverage differences between them⁵. In practice: numeric constraints (minimum, multipleOf), string length constraints (minLength) and recursive schemas tend to fall outside what the constraint enforces. The schema is accepted, the output “complies”, and nobody checked your business rule. Keep the step-2 validator even when you’re on step 3.

Not every task loses. On classification, narrowing the answer space sometimes helps: in the same study, one of the evaluated models improved on a diagnostic dataset with JSON mode enabled¹. That tracks: in classification, the format carries the answer.

Where to start

1. Measure before deciding. Run 100 real inputs through the simplest prompt and count how many parsed. If it’s 98, step 2 is enough; if it’s 60, look at what broke before climbing a step.
2. Sort the failures by kind. Markdown fences, commas, quotes and truncation have different fixes, and three of them are one-liners.
3. Write the schema with reasoning first. If the task requires thinking, the reasoning field has to come before the result field.
4. Move to constrained decoding when the cost of failure is high, not when the failure rate is high. Unsupervised batch ingestion justifies it; a screen with a human looking at it usually doesn’t.
5. Keep the validator. It stops being the primary mechanism and becomes the safety net against truncation and against the part of the schema the constraint doesn’t enforce.

Cost

Three rough estimates, to size the decision.

Validate and retry costs one extra call on the fraction of cases that fail. At a 2% failure rate with one repair pass, the overhead lands around 2% of total output cost — immaterial. At 30%, you’re paying a third more and should fix the prompt before automating the repair.

Constrained decoding costs little in generation time when the implementation precomputes its index: Willard and Louf report small overhead over unconstrained generation², and Beurer-Kellner and colleagues’ minimally invasive approach even beats unconstrained throughput thanks to speculative decoding⁴. The real cost sits in the first call for each new schema and in the work of keeping schemas stable enough for the cache to earn its keep.

Not structuring at all costs the parser you’re going to write, the edge cases you’ll discover in production, and the wrong data that reached the database before anyone noticed. That’s the only one of the three that never shows up on an invoice.

Footnotes

1. Tam et al. (2024) compare constrained decoding, format-restricting instructions and two-step conversion across reasoning and classification tasks. ↩ ↩² ↩³ ↩⁴

2. Willard and Louf (2023) recast guided generation as transitions between finite-state machine states and build an index over the model’s vocabulary, bringing per-token cost down to constant on average. ↩ ↩²

3. Geng et al. (2023) show that formal grammars describe the output space of a wide range of NLP tasks, and propose input-dependent grammars for structures that vary case by case. ↩

4. Beurer-Kellner et al. (2024) measure the accuracy loss caused by misalignment between subword tokens and the constraint, and present a subword-aligned algorithm that removes it. ↩ ↩²

5. Geng et al. (2025) evaluate six constrained decoding frameworks over 10K real-world JSON schemas, on efficiency, coverage and output quality. ↩