Asking nicely for valid JSON is not a strategy. Here is how constrained decoding actually makes bad output impossible.
Asking a model nicely to "return valid JSON" and hoping it complies is not a production strategy. Constrained decoding, also called guided decoding, is the actual engineering answer: a mechanism that makes invalid output impossible at the token level, rather than a validation step that catches bad output after the fact. This guide covers how structured output llm generation and grammar-based constrained decoding actually work under the hood, and a real tradeoff most explanations skip: the schema itself can silently degrade output quality if you're not careful about how it's structured. Several open-source libraries, including tools sometimes searched for as outlines llm implementations, apply the same underlying mechanism described here.
Key takeaways
At every step of ordinary, unconstrained generation, a model produces a probability distribution, logits, over its entire vocabulary, and the next token gets sampled from that distribution. Constrained decoding inserts one additional step into that process: before sampling happens, a constraint engine checks which tokens in the vocabulary would keep the output consistent with a target grammar, and masks every other token by setting its logit to negative infinity, effectively removing it from consideration. The model then samples only from the tokens that remain legal.
This is implemented using a state machine or grammar compiled from whatever the target constraint actually is: a JSON schema compiles into something closer to a schema walker, a regular expression compiles into a deterministic finite automaton, and a more general context-free grammar compiles into a pushdown automaton, each a different constraint specification rather than interchangeable representations of the same thing. At each generation step, that state machine reports exactly which tokens are legal continuations given everything generated so far, and the engine applies the mask accordingly. Because this happens at the sampling step rather than by modifying the model itself, the model's underlying weights and its actual competence at the task are completely unchanged; only which tokens it's allowed to choose from is restricted.
Json mode llm support, the simpler and more widely available option most managed APIs expose, generally guarantees the llm json output is syntactically valid JSON without necessarily enforcing a specific schema's fields, types, or structure beyond that. Full grammar-based constrained decoding goes further, enforcing an actual json schema llm implementation, a specific regular expression, or an arbitrary context-free grammar, guaranteeing not just that the output parses as JSON but that it matches the exact shape you defined, down to which fields are present and what type each one is.
This distinction matters in practice: JSON mode alone can still produce syntactically valid JSON that's missing a required field or has the wrong type for a value, which then fails downstream when your code tries to actually use it. Full schema-constrained decoding closes that gap at the token level, so a downstream parse failure becomes structurally impossible rather than merely less likely.
⚡ Field order is not cosmetic, where the implementation follows schema order
JSON Schema itself doesn't define property order as a semantic constraint, and different providers handle it differently: some preserve declaration order in generation, others have been observed reordering properties alphabetically depending on how the schema is serialized internally. But for structured-output systems that do generate fields in the order declared, field order is a real constraint on the model's generation process, not just a stylistic choice. Because generation is autoregressive, a schema that places a conclusion or answer field before a reasoning or explanation field forces the model to emit that answer-bearing field before it has generated the reasoning tokens that would otherwise support it. Research has found measurable reasoning degradation under some structured-generation setups, particularly when the output format forces reasoning into an unfavorable order. Placing a reasoning field before the answer field it supports lets the model generate that reasoning first, recovering most of that lost quality on systems where field order is respected; the fix costs nothing beyond reordering the schema.
Guaranteeing that output parses correctly is a genuinely different guarantee from the output being correct, and it's worth being precise about that distinction. Constrained decoding enforces shape, not truth: a required, non-nullable field the source content simply doesn't support still has to be filled with something, since the grammar has no legal token sequence representing "I don't know" or "this field isn't applicable here." The model fills it anyway, confidently, and the result parses perfectly while being factually invented.
The practical fix is designing the schema to encode the actual uncertainty states an application can encounter, rather than making every field mandatory by default: allowing a field to be null, including an explicit "not found" or "unknown" enum value, or adding a companion confidence or source field alongside a value that might be uncertain. Deeply nested schemas with many optional branches and long enumerated value lists also generally perform worse than flatter, simpler ones, since more grammar states generally means more masking decisions and more opportunities for the constraint to push the model toward unfamiliar token sequences.
Constrained decoding is now supported as a serving-engine feature across several major open-source inference engines, not just as a proprietary feature locked to one specific structured output api, which means the underlying structured generation llm mechanism works similarly whether you're calling a hosted endpoint or self-hosting a model with a modern serving stack. This same llm grammar mechanism is also what makes function calling json schema arguments reliable, since a tool call's arguments are themselves a structured output constrained to the schema the tool defines. packet.ai's Token Factory is being built as an OpenAI-compatible endpoint, meaning structured output requests use the same familiar JSON schema interface regardless of which specific model handles the request. It's currently in private preview, with the specific model catalog and pricing still being finalized.
Join the waitlist for early access once it opens.
Last reviewed: September 18, 2026. Constrained decoding libraries and provider support evolve quickly; verify current capabilities against your specific serving engine or API's documentation. For the broader mechanics of how token generation works, see the packet.ai What Is LLM Inference guide.
Same models. Same API. Fraction of the cost. Start free — no credit card required.
Start Building →
