How do you constrain what the model can generate?

You describe the format as a formal grammar and the inference engine enforces it while generating: at each token an automaton works out which vocabulary entries keep the sequence valid and zeroes the probability of every other one before sampling. The model is not persuaded to follow the format. It becomes incapable of leaving it.

This is the machinery underneath the JSON mode and the strict schema described in how to force valid JSON. The difference here is that you write the grammar instead of letting the provider derive one from your schema, which buys you formats JSON Schema cannot describe.

Two things are worth separating before going further. Constraining decoding is not validating afterwards, which belongs to JSON Schema and Zod. And it is not cutting generation at a marker, which is what stop sequences do: a stop sequence trims the end, a grammar governs every token from the first one.

The mechanism, one token at a time

The model generates one token at a time, and each step produces one number per vocabulary entry, the logit. Normally those numbers become probabilities and one token gets sampled. Constrained decoding wedges itself between those two steps.

Before generation starts, the grammar is compiled into an automaton. During generation, the state of that automaton after the tokens emitted so far determines the set of tokens acceptable right now. After {"qty": that set holds digits and a minus sign, and does not hold a quote or the word twelve. The engine replaces the logit of every token outside the set with minus infinity, and only then normalises and samples.

Notice what that implies: the constraint never looks at the meaning of the answer and knows nothing about the prompt. All it knows is where the sequence sits inside the grammar. That is the strength and the ceiling of the technique at once.

Why invalid becomes unreachable

The order of operations is what gives the guarantee. Minus infinity through the softmax’s exponential becomes exactly zero, so an illegal token gets probability zero rather than zero-point-something. No value of temperature, top-k or top-p changes that: all three operate on a distribution that token already left.

The guarantee is therefore structural, not statistical. It is not that the model started making fewer mistakes; it is that the invalid sequence stopped existing in the output space. The sentence works in reverse too: if your grammar allows something, the model can produce it, however absurd it is in context.

The vocabulary problem

The obvious objection is cost. Testing the whole vocabulary at every token is expensive when the vocabulary holds tens or hundreds of thousands of entries, and a naive implementation walks all of it at every step, for every request running in parallel.

Willard and Louf solved the regular version of the problem by inverting the computation. Instead of asking “which tokens fit here” during generation, they pre-compute, once, an index associating each automaton state with the set of tokens acceptable in that state. At generation time the lookup is a read, and the per-token cost is constant on average¹.

XGrammar went after the context-free version, which is the one real JSON demands. The core idea is splitting the vocabulary in two classes: tokens whose validity does not depend on the stack state, which can be checked once, and tokens that depend on context and must be interpreted at runtime. They add a persistent stack to speed up the second class and overlap grammar computation with GPU execution. The reported result reaches 100 times the speed of earlier solutions, with near-zero overhead in end-to-end generation².

That engineering, rather than any change in the models, is why constrained decoding moved from paper to API parameter between 2023 and 2025.

Regular or context-free

The grammar class you pick decides what you can express and what compilation costs.

Regular grammars handle fixed-structure formats: a date, an enum, an identifier with a prefix, a number with decimal places. It is what a regular expression describes, and it is the cheapest to compile and to index.

Context-free grammars become necessary the moment nesting is unbounded — and JSON is, since an object can hold a list of objects with no depth limit. Counting open braces requires a stack, and a stack is exactly what a finite automaton lacks. Engines limited to finite automata approximate this by fixing a maximum depth, which works until the day your document has one level more.

In practice the most direct way to write a grammar yourself is llama.cpp’s GBNF: a text file with rules in a BNF variant, passed alongside the prompt. It shows up whenever someone serves an open-weight model and needs a format no provider offers. A minimal grammar for a verdict with a justification fits in four lines:

root    ::= "REASON: " reason "\nVERDICT: " verdict
reason  ::= [^\n]+
verdict ::= "approved" | "rejected" | "undecided"

Look at what those four lines guarantee and what they do not. They guarantee the last word is one of three and that some text precedes it, and they guarantee that with no instruction in the prompt at all. They do not guarantee the justification has anything to do with the verdict.

One cheap variation deserves a mention because it is rarely used: constrain only part of the output. Letting the model write freely while it reasons and switching the grammar on only for the closing stretch recovers most of the quality that constraining costs, while still guaranteeing the shape your code will consume. Whoever serves their own model does this by swapping grammars mid-generation; on commercial APIs, the equivalent is a reasoning field declared ahead of the result field.

Beyond JSON

The technique was born far from JSON. PICARD, in 2021, constrained decoding for text-to-SQL models through incremental parsing: at each step, tokens that would render the query invalid were rejected. That turned fine-tuned T5 models with passable performance into the best-scoring solutions of the moment on Spider and CoSQL³. The gain came from removing known errors from the output space, not from more training.

Geng and colleagues generalised the argument in 2023: formal grammars describe the output space of a far wider range of tasks than parsing and code generation, and constrained decoding works as a unified framework for structured NLP tasks generally. They introduced input-dependent grammars, letting the grammar change per case, and evaluated on information extraction, entity disambiguation and constituency parsing, with constrained models beating unconstrained ones and, in some cases, models fine-tuned for the task⁴.

The recurring pattern: whenever the correct output space is describable and much smaller than the possible output space, constraining beats instructing.

Where it fails

Token splits leave familiar territory. This is the least intuitive failure and the most expensive. The model learned a distribution over token sequences during training, including how words tend to be split. The mask can force a split that never occurs naturally: if the most likely token was "12 and the mask only permits ", the model continues from a prefix that is rare in its own distribution. Beurer-Kellner and colleagues measured the damage on a JSON-encoded GSM8K with Mistral 7B: 41.5% correct unconstrained, 30.8% with a naive constraint and 41.8% with a subword-aligned one⁵. The entire loss came from the implementation, not from the idea.

Greedy masking distorts the distribution. Even with subword alignment solved, zeroing tokens at each step is not the same as sampling from the model’s distribution conditioned on the grammar. Park and colleagues showed that constrained decoding produces grammatical outputs with likelihoods out of proportion to what the model would assign, and proposed sampling that accounts for an approximation of each prefix’s future grammaticality, yielding higher-likelihood outputs without giving up the constraint⁶. The intuition is local: the mask picks the best token now without knowing whether that path leads into a corner where only poor continuations remain.

Specification coverage is partial. When the grammar is derived from a JSON Schema, part of the schema never becomes a constraint. A 2025 benchmark evaluated six frameworks over 10,000 real-world schemas and found meaningful coverage differences between them⁷. Numeric bounds, minimum string length and recursion are the usual casualties. The practical effect is bad: the schema is accepted, the output “complies”, and the rule you wrote was checked by nobody.

Valid is not correct. The grammar guarantees shape. It has no idea whether the identifier exists in your database, whether the chosen category is the right one, or whether the number makes sense. Constraining trades a loud failure for a quiet one, and that trade only pays if content evaluation sits on the other side.

The grammar becomes a second source of truth. Whatever you write in BNF already exists somewhere in your codebase, as a type, a database column or a validator. Now it exists twice, in two languages, maintained by hand. The drift is quiet and predictable: someone adds a fourth verdict value to the enum in the application, the grammar keeps three, and the model becomes structurally unable to emit the new one. Nothing errors out — the output is still grammatical, just never the value you added. Generating the grammar from the same declaration that produces your types is more work up front and the only version that survives a year of changes.

Compilation costs, and it costs up front. A large schema becomes a large automaton, and construction happens on the first call. Regenerating the grammar on every request, with rule order shifting, throws the compilation cache away and turns a one-time cost into a per-call one.

Truncation survives. Nothing in the mask stops a response ending mid-object at the token limit. The result is a grammatically perfect and unusable prefix, so checking the stop reason stays mandatory.

What it costs

In generation time, little, provided the implementation pre-computes its index. Willard and Louf’s approach reduces per-token work to constant cost on average¹, and Beurer-Kellner and colleagues’ minimally invasive version can even beat unconstrained generation on throughput, because the mask narrows the space enough for speculative decoding to hit more often⁵. XGrammar reports near-zero end-to-end overhead².

The real cost sits in three less visible places: compiling the first call of each new grammar, the work of writing and maintaining the grammar, and the quality lost when the library is naive. All three are engineering costs, not line items.

When it is worth it

1. Use it when the cost of one malformed output is high, not when the malformation rate is high. Unreviewed batch ingestion justifies it; a screen with a human on it rarely does.
2. Prefer the provider’s strict schema while it holds. Writing a grammar only pays off when the format is not JSON or the rule does not fit in JSON Schema.
3. Write the loosest grammar that solves the problem. Every extra rule is another chance to force an odd token split and drag quality down.
4. Put the reasoning field ahead of the result field. The model writes in grammar order, and result before justification removes the justification.
5. Measure with and without the constraint on the same task. It is the only way to know whether your library sits at 41.8% or at 30.8%.
6. Keep the validator. It covers truncation and the slice of the schema the grammar never expressed.

Footnotes

1. Willard and Louf (2023) pre-compute an index mapping automaton state to the set of acceptable tokens, making per-token cost constant on average instead of proportional to the vocabulary. ↩ ↩²

2. Dong et al. (2024) separate context-independent tokens, checkable once, from stack-dependent ones, use a persistent stack, and overlap grammar computation with GPU execution; they report up to 100 times the speed of earlier solutions. ↩ ↩²

3. Scholak et al. (2021) constrain decoding through incremental parsing, rejecting inadmissible tokens at each step, and take fine-tuned T5 models to the best results of the moment on Spider and CoSQL. ↩

4. Geng et al. (2023) argue that formal grammars describe the output space of a wide range of NLP tasks, introduce input-dependent grammars, and evaluate on information extraction, entity disambiguation and constituency parsing. ↩

5. Beurer-Kellner et al. (2024) attribute the accuracy loss to misalignment between the mask and the model’s subwords, and present an aligned algorithm that removes it while improving throughput through speculative decoding. ↩ ↩²

6. Park et al. (2024) show that constrained decoding distorts the model’s distribution, producing grammatical but low-likelihood outputs, and propose a sampler that approximates the future grammaticality of prefixes to correct it. ↩

7. Geng et al. (2025) evaluate six constrained-decoding frameworks over 10,000 real-world JSON Schemas, on efficiency, coverage and quality. ↩