Structured Outputs: Making LLMs Speak JSON Reliably

Every LLM feature in production eventually hits the same wall: the model returns something that almost looks like JSON, your parser throws, and a user stares at a spinner that never resolves. Free text is a demo format. The moment a machine consumes the output, structure stops being a nice-to-have and becomes the contract.

Why "just ask for JSON" fails

Prompt a model with "respond only in JSON" and it will comply most of the time. The failures are what kill you:

• Markdown code fences wrapped around the object, so JSON.parse chokes on the first backtick.
• A polite preamble ("Here is the JSON you requested:") before the payload.
• Trailing commas, single quotes, JavaScript-style comments, unescaped newlines inside strings.
• Prose appended after the closing brace ("Let me know if you need anything else!").
• The right syntax with the wrong shape: renamed keys, a string where you expected a number, an extra field your code never asked for.

In our experience, naive prompting fails to parse on a low single digit percentage of requests. At ten requests a day that is invisible. At fifty thousand requests a day it is a permanent background incident, and regex-based "JSON extraction" utilities just move the failure somewhere harder to debug.

The deeper problem: parse failures are the failures you can see. Schema drift is silent. A field that quietly switches from "priority": 2 to "priority": "2" will not throw at the parser. It will throw three services downstream, at 2 a.m.

The reliability ladder

There are four levels of output reliability. Know which one you are on, and which one your use case actually requires.

Level 1: Prompt and pray

You describe the format in the prompt, add a few-shot example, and parse whatever comes back. Acceptable for prototypes and internal scripts. Not acceptable for anything with an SLA.

Level 2: JSON mode

Most hosted APIs offer a switch that guarantees syntactically valid JSON. That eliminates fences, preambles and trailing prose. It guarantees nothing about your schema: the model can still return valid JSON with the wrong keys, wrong types, or a creative structure of its own invention. JSON mode is a floor, not a solution.

Level 3: Schema-enforced APIs

The current state of the art on hosted models. You attach a JSON Schema to the request and the provider enforces it during generation. On the Claude API this is output_config.format with a json_schema type, or strict: true on a tool definition. OpenAI has an equivalent under response_format. Under the hood the provider compiles your schema into a grammar and constrains token sampling against it, so a response that violates the schema is not merely unlikely, it is unreachable.

Two operational details matter here. First, new schemas usually incur a one-time compilation cost on the first request, then hit a server-side cache (around 24 hours on the Claude API), so keep schemas stable rather than generating them dynamically per request. Second, enforcement has documented limits: recursive schemas, numeric bounds like minimum, and string constraints like maxLength are typically unsupported or stripped. Those constraints still belong in your validation layer, just client-side.

Level 4: Constrained decoding on self-hosted models

Running open-weight models on your own infrastructure? You own the sampler, which means you can enforce any grammar you want. Libraries like Outlines, XGrammar and llguidance integrate with vLLM and TensorRT-LLM; llama.cpp ships GBNF grammars. This is the strongest guarantee available, and it extends beyond JSON: you can constrain output to a regex, a SQL dialect subset, or a custom DSL.

How constrained decoding actually works

This is worth understanding because it explains both the guarantee and its blind spots.

An LLM generates one token at a time by producing a probability distribution over its vocabulary. Constrained decoding compiles your schema or grammar into a finite state machine, tracks which state the generation is in, and masks the logits of every token that would lead to an invalid continuation before sampling. Invalid tokens get their probability forced to zero. The model literally cannot emit a token that breaks the grammar.

The engineering difficulty is that token boundaries do not align with characters. A single token might contain ": or },{". Efficient engines precompute token-level automata so the mask is a lookup rather than a per-step scan of a 100,000-entry vocabulary. This is the difference between XGrammar-class performance and a naive implementation that halves your throughput.

The critical insight: constrained decoding guarantees syntax, never semantics. The model will always produce a schema-valid object. Whether the object is true is an entirely separate question.

Tool calling is structured output with a decision attached

Teams conflate function calling and structured outputs constantly. The distinction is simple:

• Use tool calling when the model should decide whether and which action to take: search the CRM, send an email, escalate a ticket. The schema constrains the arguments; the model owns the decision.
• Use response formatting when you always want one document back: an extraction, a classification, a report. No decision, just shape.

The common anti-pattern is a single mega-tool named respond with a 40-field schema, used for everything. You lose the decision signal, you bloat every request with schema tokens, and you make the hardest fields compete with the easiest ones. Split by intent. And when you do define tools, turn on strict mode: on the Claude API that means strict: true on the tool plus additionalProperties: false and a required array in the schema, which guarantees the arguments validate exactly. This matters most in agent loops, where one malformed argument can derail an entire multi-step run. We covered that failure class in our guide to building AI agents for business.

Designing schemas the model can actually hit

Schema design is prompt engineering. Every rule here comes from watching real pipelines fail.

• Flat beats deep. Two levels of nesting is fine. Five levels invites confusion and burns tokens on structure instead of content.
• Enums over free strings. "category": {"enum": ["billing", "technical", "account", "other"]} is testable. A free string field is a bug generator.
• Describe every field. The description keys are read by the model. Treat them as prompt real estate: state units, formats and edge-case behavior.
• Make absence explicit. Prefer "anyOf": [{"type": "string"}, {"type": "null"}] over optional keys. Forced-required fields with no null option teach the model to invent values.
• Order fields deliberately. Generation is autoregressive: the model writes fields in schema order. Putting a short reasoning field before a classification label lets the model think before it commits, and in our experience that reliably improves label quality on ambiguous inputs. It costs output tokens, so use it where accuracy pays for it.

A schema that has survived production, for support ticket triage:

{
  "type": "object",
  "additionalProperties": false,
  "required": ["reasoning", "category", "priority", "needs_human"],
  "properties": {
    "reasoning": {
      "type": "string",
      "description": "One or two sentences justifying the triage decision."
    },
    "category": {
      "type": "string",
      "enum": ["billing", "technical", "account", "other"]
    },
    "priority": {
      "type": "integer",
      "enum": [1, 2, 3],
      "description": "1 = urgent, 3 = routine."
    },
    "needs_human": { "type": "boolean" }
  }
}

Failure modes nobody warns you about

Schema enforcement removes parse errors. It introduces a subtler class of problems:

• Schema-valid garbage. An empty string satisfies "type": "string". Watch for "", "unknown" and suspiciously default-looking values; they are the new null pointer.
• Truncation. If generation hits max_tokens mid-object, you get a valid JSON prefix that is not valid JSON. Always check the stop reason before parsing, and size max_tokens with headroom.
• Forced-choice hallucination. An enum with no escape hatch forces the model to pick a wrong answer confidently when none of the options apply. Always include an other or unknown variant, and monitor how often it fires.
• Refusals. Safety systems can decline a request; the refusal will not match your schema. Handle the refusal stop reason as a first-class branch, not an exception path.
• Grammar pressure on reasoning. Hard constraints can force the model down a syntactic path before it has "decided" the content, which measurably hurts quality on hard tasks. Mitigations: the reasoning-field trick above, or a two-step pipeline where step one reasons in free text and a cheap second call formats it.
• Streaming. A partially streamed object is unparseable until the final brace. If your UI renders progressively, use an incremental JSON parser that yields partial objects, and treat everything as tentative until the stream ends.

Extraction-heavy systems feel these hardest, because extraction runs at high volume over messy inputs. If you are building retrieval pipelines, the same discipline applies to every metadata and chunk-labeling step; see our breakdown of RAG systems.

The validate-and-repair loop

Even with server-side enforcement, run client-side validation. Defense in depth, plus a home for the constraints the provider cannot enforce (lengths, ranges, cross-field rules). Define the schema once in code and derive the JSON Schema from it: Zod in TypeScript, Pydantic in Python. Never maintain the schema in two places.

const result = TicketSchema.safeParse(JSON.parse(raw));
if (!result.success) {
  // One repair attempt: resend with the validator errors attached
  const repaired = await llm.complete({
    messages: [...original, assistant(raw), user(
      `Your output failed validation:\n${formatZodError(result.error)}\nReturn a corrected object only.`
    )],
  });
  return TicketSchema.parse(JSON.parse(repaired));
}

Rules for the loop: cap retries at one or two (past that you are burning money on a model that has decided to be wrong), log the raw output alongside the parsed result on every request, and alert on retry rate. A rising repair rate is your earliest signal that a prompt, model version or input distribution shifted.

A deployment checklist

Before a structured-output feature ships, walk this list:

1. Decide the level. Machine-consumed output gets Level 3 or 4. No exceptions for "it usually works".
2. Define the schema in Zod or Pydantic and generate the JSON Schema from it.
3. Enable strict enforcement: schema-constrained response format, or strict: true tools with additionalProperties: false.
4. Add an unknown escape hatch to every enum, plus a monitored counter on it.
5. Check stop reasons (truncation, refusal) before parsing anything.
6. Validate client-side and wire the single-retry repair loop with error feedback.
7. Log raw outputs, parse failures, retry counts and per-field anomaly rates.
8. Version the schema like a public API. A schema change is a deploy: run it through your eval set first.
9. Load-test the first-request schema compilation path so a cold cache does not spike your p99.

One more operational note: schemas and field descriptions are input tokens on every single call, and repair retries multiply output cost. Structured outputs interact directly with your token bill, and the levers are the same ones we documented in our LLM cost optimization guide: trim schemas, cache aggressively, route easy extractions to smaller models.

How Innovation T can help

Innovation T designs and ships LLM systems where the output contract is engineered, not hoped for: schema-enforced extraction pipelines, strict-mode agent tooling, validation and repair layers, and the observability to prove reliability week after week. We have done this across support automation, document processing and internal tooling, on hosted APIs and self-hosted stacks alike.

If your team is fighting flaky JSON or planning its first production LLM feature, explore our software and AI engineering services or contact us for a technical assessment. We will tell you exactly which level of the ladder your use case needs, and build it.