Proposal

Add a new node Structured Outputs / Constrained Decoding to the Inference Engineering roadmap.

Why it belongs

Structured output is how LLM output reliably becomes machine-consumable: instead of free prose, generation is constrained to a schema (JSON schema, regex, grammar) at decode time, so downstream code can parse it with no repair pass. Major inference engines ship this as a first-class feature (vLLM, SGLang), and there is a dedicated open-source layer for it (Outlines, Guidance). Products are being built around it as the core primitive: e.g. TypeSafe AI's Jev returns typed decisions with probabilities instead of prose. An inference engineer who cannot explain how constrained decoding works, what it costs, and when it changes output quality has a gap in exactly the area this roadmap covers.

Suggested placement

As a sibling of Speculative Decoding: both are decode-time techniques that change how tokens come out, one for speed, one for shape. It sits naturally after Prefill and Decode Phases (it only constrains the decode loop) and before Client Integration (the typed output is the contract the client consumes).

Suggested node content

Structured Outputs / Constrained Decoding

Constrained decoding masks logits at each sampling step so the model can only emit tokens that fit a schema, such as a JSON schema, regex, or grammar. This makes model output directly parseable by downstream code with no repair pass. It costs latency up front: compiling the grammar adds to time-to-first-token. And a constraint guarantees the output's shape, not its correctness: when the model's best answer does not fit the schema, forcing it to fit can degrade quality. End-to-end it usually wins anyway, because a typed decision is a handful of tokens where prose would be hundreds, and decode time dominates. So it is an engineering tradeoff like any other inference technique.

Visit the following resources to learn more:

• @official@vLLM Structured Outputs
• @official@SGLang Structured Outputs
• @official@Jev: TypeSafe AI's decision model
• @opensource@Outlines: structured generation
• @opensource@Guidance: guided generation