How do you guarantee the model returns valid JSON?

You have four rungs, cheapest to strongest: ask for the format in the prompt, turn on the provider’s JSON mode, hand over a strict schema, or compile your own grammar. Only the last two guarantee anything about the fields. None of them stops truncation at the token limit, so the validator stays necessary even at the top of the ladder.

This is the hands-on half of structured outputs: which switch to flip, what each one leaves out, and how to clean up the rest. Declaring the shape and checking the values belongs to JSON Schema and Zod.

The fourth rung, writing the grammar yourself, has its own article in grammar-constrained decoding; here it only appears as the ceiling. One thing worth settling first: format instructions belong in the system message, and the three roles are not interchangeable for this. A format rule repeated in the user message competes with the actual input instead of reinforcing itself.

Where JSON breaks

Before climbing a rung it helps to know what you are climbing away from. Each failure has a different cause and a different repair, and three of them are solved without touching the prompt at all.

The markdown fence is the common one. The model returns the right object wrapped in ```json, often with a polite sentence in front. The cause is statistical: it saw far more JSON inside code blocks than JSON standing alone. Repair: slice between the first brace and the last.

The trailing comma is a stray , before a } or a ]. It shows up when the field list is long and the model loses track of which item was the last one. Repair: strip it before parsing, or move up to rung two.

Quotes break in two ways that share nothing but a name. Curly quotes instead of straight ones happen when the field content came from formatted text. Straight quotes left unescaped inside a string close the value halfway and knock everything after it out of alignment. The second is nastier, because the parser fails at a position with no relationship to the cause.

Truncation is the one no text surgery reaches. The JSON is correct as far as it got, and it stopped because it hit the token limit. The symptom is failing only on long cases, which reads as randomness until you look at the response’s stop reason and find the limit instead of a natural ending.

Comments are the rarest and the easiest to misdiagnose. The model writes // optional beside a field, usually because your prompt described the schema in prose with annotations like that. JSON has no comments, so the parser refuses. The real repair is upstream: take the comments out of the example you sent.

The ladder

The four rungs are not competing options. They are layers, and each one closes a class of error the previous one left open. The most common design mistake is jumping straight to the top for safety, paying the complexity, and still shipping without a validator — because no rung solves the problem that actually matters, which is whether the value inside the field is correct.

Rung 1: ask in the prompt

Cheap, immediate, and enough to find out whether the task works at all. Three things move the hit rate for real, and writing the instruction in capitals is not one of them.

Show the output, don’t describe it. One worked example communicates field names, types, date format and nesting depth in a single stroke. Prose spends more tokens and says less.

Say what must not appear. “Reply with the JSON object only, no code fence and no text before or after” removes most fences. It is one of the few negative instructions that pays here.

Leave headroom in the token limit. Size it for the worst case, not the median one. A list that usually carries three items and one day carries forty is the classic truncation setup.

And an architectural detail that matters more than it looks: if the schema is fixed, it belongs in the system message, which is the stable part of the prompt. That leaves the user message carrying only the real input and keeps the prefix byte-identical between calls, which is the precondition for prompt caching to do anything.

Prefill, where it survives

Prefill is the cheapest trick on rung one. Instead of ending the message list with the user, you append an assistant message containing nothing but { and send that. The model continues from where the message stopped, and it can no longer open a markdown fence, because the fence would have to come before a brace that is already written.

Two traps. First, the brace you wrote does not come back in the response. You get the continuation, and your code has to glue the { back on before parsing. Second, not every API allows it: several providers now require the last message to come from the user and return a 400 otherwise. As of July 2026 that already covers part of the newer commercial models, while open-weight models served locally still accept it without complaint. Where prefill is gone, the answer is not a firmer instruction — it is the next rung.

Rung 2: JSON mode

JSON mode is a parameter on the call that constrains decoding to produce syntactically valid JSON. It is not a stronger instruction and not a post-processing step: from that point on, tokens that would lead to an impossible sequence get zero probability before sampling.

What it guarantees is exactly one thing: the text opens in a parser. All four syntax failures from figure 1 disappear, comments and trailing commas included, because none of them is legal JSON.

What it does not guarantee is everything else. The shape is free, so {"answer": "I don't know"} satisfies JSON mode just as well as the object you were expecting. Missing fields, invented field names, wrong types and out-of-range enum values all survive. So does truncation, because the token limit sits upstream of the constraint.

One consequence catches people off guard: with JSON mode on and a prompt that never says clearly what to produce, models sometimes emit an empty object or run into a long stretch of whitespace until they hit the limit. The constraint guarantees the format and does not replace the instruction.

Rung 3: strict schema

Here you send the JSON Schema with the call and the provider compiles it into a constraint over decoding. The difference from rung two is that the constraint now knows your fields: after {" only prefixes of field names that exist in the schema get through, after "qty": only digits do, and a three-value enum makes a fourth value unreachable.

The obvious objection is the cost of testing the whole vocabulary at every token. Willard and Louf handled that by building, before generation starts, an index mapping each automaton state to the set of tokens acceptable there, which makes the per-token work constant on average¹. XGrammar pushed it further by separating tokens that can be pre-checked once from tokens that depend on the stack context, and by overlapping grammar computation with GPU execution; the authors report up to 100 times the speed of earlier solutions and near-zero overhead end to end². That is why a strict schema stopped being a performance decision.

Three operational details that surface in production:

1. The schema has to be closed. Providers typically require additionalProperties: false plus an explicit list of required fields. A schema that tolerates extra keys does not become a useful constraint.
2. The first call with a new schema is slower. Compilation happens once and is cached on the provider’s side. Building the schema on the fly with the key order shifting between calls throws that cache away.
3. Specification coverage is partial. A 2025 benchmark evaluated six constrained-decoding frameworks over 10,000 real-world schemas and found meaningful coverage differences between them³. Numeric bounds, minimum string length and recursive schemas are the first things to drop out. In practice the schema is accepted, the output “complies”, and nobody checked your rule.

Rung 4: your own grammar

The top rung uses directly the machinery the other two wrap: you write the grammar, and the inference engine applies it over the logits. That is what llama.cpp’s GBNF and the guided-generation libraries expose when you serve the model yourself.

It earns its keep in three situations: the format is not JSON, the rule is context-sensitive in a way JSON Schema cannot express, or you need a guarantee over a format your provider does not offer. Outside those, rung three delivers the same thing with less code to maintain.

The slice that handles most cases

If you stayed on rung one, this is the only code that matters. It handles code fences, prose before and prose after in one move, with no case analysis:

function extract(raw: string) {
  const i = raw.indexOf('{');
  const j = raw.lastIndexOf('}');
  if (i < 0 || j < i) throw new Error('no object in the response');
  return JSON.parse(raw.slice(i, j + 1));
}

Check the stop reason before slicing. If the response ended at the token limit, the object is incomplete and no text repair will save it — what you want there is a retry with a larger limit, not a hand-rolled attempt to close the braces.

When parsing fails anyway, send the validator’s error back to the model together with its previous answer, instead of replaying the same prompt and hoping. A repair pass like that is cheap and measurable: in the study by Tam and colleagues, using a small model to reformat outputs that failed to parse improved the final result in both JSON and YAML⁴.

Where this fails

Valid syntax is not a correct answer. This is the expensive confusion. With a strict schema, {"category": "billing"} parses cleanly and the value is in the enum; none of that says the ticket was about billing. Climbing the ladder trades a loud failure for a quiet one, and without content evaluation the quiet kind goes unnoticed for far longer.

Field order changes the answer. The model writes fields in the order the schema lists them. If the result field comes before the reasoning field, there is no reasoning: the answer was already written by the time the justification starts. Tam and colleagues found this pattern consistently under JSON mode, and the fix is to declare the reasoning first⁴.

A badly implemented constraint costs accuracy. Beurer-Kellner and colleagues measured a JSON-encoded version of GSM8K with Mistral 7B and five examples: 41.5% correct unconstrained, 30.8% with a naive constraint, and 41.8% with a subword-aligned one⁵. The cause is the mask forcing a token split the model never saw in training. Treat it as evidence that the library matters as much as the decision to constrain.

The parser is rarely the culprit when quality drops. In the same work by Tam and colleagues, one model had a 0.148% parse-error rate on a task and still showed a 38.15% performance gap against free-text output⁴. If structuring made your results worse, look at field order and at the room the format took away from reasoning, not at the parse failure rate.

Cost

Orders of magnitude, enough to size the decision.

Staying on rung one with a repair pass costs one extra call on the fraction that fails. At 2% failure the increment lands around 2% of output cost, which is noise. At 30% failure you are paying a third more to automate a problem the prompt would have solved.

Moving to a strict schema costs almost nothing in generation time in current implementations¹². The real cost is the compilation on the first call with each new schema, plus the discipline of keeping schemas stable enough for the cache to help.

Where to start

1. Run 100 real inputs on rung one and count how many parsed. Without that number you are picking a rung by feel.
2. Sort the failures by figure 1. If most are markdown fences, the slice fixes it today. If most are truncation, the problem is your token limit and no rung helps.
3. Turn on the strict schema when the cost of one bad record is high, not when the failure rate is high. Unsupervised batch ingestion justifies it; a screen with a human looking at it usually does not.
4. Declare reasoning before the result on any task that requires thinking.
5. Keep the validator at the top of the ladder. It stops being the main mechanism and becomes the net against truncation and against the slice of the schema the constraint never covered.

Footnotes

1. Willard and Louf (2023) recast guided generation as transitions between finite-automaton states and pre-compute an index over the model’s vocabulary, making the per-token cost constant on average. ↩ ↩²

2. Dong et al. (2024) split the vocabulary into context-independent tokens, checkable once, and stack-dependent tokens, and overlap grammar computation with GPU execution; they report up to 100 times the speed of earlier solutions. ↩ ↩²

3. Geng et al. (2025) evaluate six constrained-decoding frameworks over 10,000 real-world JSON Schemas, on efficiency, coverage and quality. ↩

4. Tam et al. (2024) compare constrained decoding, format instructions and two-step conversion, and measure the effect of field order and of a reformatting pass. ↩ ↩² ↩³

5. Beurer-Kellner et al. (2024) measure the accuracy loss caused by misalignment between the mask and the model’s subwords, and present an aligned algorithm that removes it. ↩