LLM Structured Outputs: JSON Mode, Schemas, Retries

Who this is for: developers wiring an LLM into a real product — a form-filler, a data extractor, an agent that calls your API — who are tired of JSON.parse throwing on the fifth request out of a hundred. If you need the model to return the same shape every time, this is that guide.

LLM structured outputs are the difference between a demo and a feature. A chatbot that free-types text is fine for a chat window; the moment you need to write that output into a database column, call a function with it, or render it in a UI component, you need guarantees about its shape. This guide covers the four techniques that actually get you there in 2026: JSON mode, JSON Schema-constrained decoding, validate-and-retry loops, and streaming partial JSON — plus where each one breaks.

Why "just ask for JSON" fails

Prompting "respond only in JSON" gets you to roughly 90-95% valid JSON with a capable model like Claude Opus 4.5 or GPT-5. That sounds good until you do the math: at 10,000 requests a day, a 3% failure rate is 300 broken responses daily, each one either crashing a pipeline or silently corrupting data downstream. Common failure modes:

• Markdown code fences wrapped around the JSON (``json ... ``) even when told not to
• Trailing commentary before or after the object ("Here's the JSON you asked for:")
• Valid JSON with the wrong shape — a string where you expected a number, a missing required key
• Truncated output when the response hits a token limit mid-object

None of these are exotic edge cases. They show up in week one of any real integration. The fix isn't a better prompt — it's not relying on the prompt alone.

JSON mode vs. schema-constrained decoding

There are two distinct levels of guarantee, and vendors conflate them in marketing copy. Know the difference.

JSON mode alone still lets the model invent field names or skip a required key because it's only constraining syntax, not structure. Schema-constrained decoding constrains the actual token sampling against your schema's grammar — the model literally cannot emit a token that violates the schema. If your provider offers strict schema mode, use it; it's the single highest-leverage fix available.

Tip: with Claude, the practical equivalent of schema-constrained decoding is forcing a tool call with tool_choice: {"type": "tool", "name": "extract_data"} and defining your shape as the tool's input_schema. The model is never asked to "write JSON" — it's asked to call a function, and function-calling is where these models are most reliable. See LLM tool calling in depth if this is new to you.

Designing the schema itself

A schema that's technically valid JSON Schema can still produce unreliable outputs if it's badly shaped for a language model. Rules that hold up in production:

1. Keep it flat where you can. Deep nesting (4+ levels) increases the chance the model loses track of which object it's populating.
2. Every field needs a description. Models use these as instructions, not just documentation. "description": "ISO 8601 date, e.g. 2026-07-14" beats a bare "type": "string" every time.
3. Use enum aggressively. If a field can only be one of five values, constrain it. This is free accuracy — you're removing an entire failure mode.
4. Mark almost everything required. Optional fields tempt the model to omit things inconsistently. If a field can legitimately be absent, use nullable types instead and require the key to exist with an explicit null.
5. Avoid oneOf/anyOf unions when a discriminated object works. Add a "kind": "invoice" | "receipt" literal field and branch in your own code rather than asking the model to pick between two loosely related shapes.

{
  "type": "object",
  "properties": {
    "invoice_number": { "type": "string", "description": "Exact string as printed, no normalization" },
    "issue_date": { "type": "string", "description": "ISO 8601 date, e.g. 2026-07-14" },
    "currency": { "type": "string", "enum": ["ILS", "USD", "EUR"] },
    "total_amount": { "type": "number", "description": "In minor units are NOT used; plain decimal, e.g. 1499.90" },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "properties": {
          "description": { "type": "string" },
          "quantity": { "type": "integer" },
          "unit_price": { "type": "number" }
        },
        "required": ["description", "quantity", "unit_price"]
      }
    }
  },
  "required": ["invoice_number", "issue_date", "currency", "total_amount", "line_items"]
}

The validate-and-retry loop

Even with strict schema mode, treat the model's output as untrusted input from a third party — because that's what it is. A production extraction pipeline looks like this:

import { z } from "zod";

const InvoiceSchema = z.object({
  invoice_number: z.string().min(1),
  issue_date: z.string().regex(/^\d{4}-\d{2}-\d{2}$/),
  currency: z.enum(["ILS", "USD", "EUR"]),
  total_amount: z.number().positive(),
  line_items: z.array(z.object({
    description: z.string(),
    quantity: z.number().int().positive(),
    unit_price: z.number().nonnegative(),
  })).min(1),
});

async function extractInvoice(text, attempt = 1) {
  const raw = await callModel(text); // schema-constrained call
  const parsed = InvoiceSchema.safeParse(raw);

  if (parsed.success) return parsed.data;

  if (attempt >= 3) {
    throw new Error(`Extraction failed after 3 attempts: ${parsed.error.message}`);
  }

  // Feed the validation error back to the model as correction context
  return extractInvoice(
    `${text}\n\nYour previous attempt failed validation: ${parsed.error.message}. Fix it.`,
    attempt + 1
  );
}

Three things matter in that loop:

• Validate with a real schema library (Zod, Pydantic, Ajv) — not a hand-rolled set of if checks that will miss edge cases you haven't thought of yet.
• Feed the validator's error message back to the model. "Field total_amount must be positive, got -12" is a far better correction prompt than re-asking the same question.
• Cap retries at 2-3. If a well-designed schema plus one correction round can't get a valid response, the problem is usually the input document, not the model — log it and route to a human rather than retrying forever and burning tokens.

Warning: don't silently swallow validation failures and return a default/empty object. That's how a broken PDF quietly becomes a zero-dollar invoice in your database. Fail loud, log the raw output, alert.

Streaming partial JSON

For anything user-facing — a live-filling form, a progressively rendered summary card — you don't want to wait for the full response before showing anything. Streaming partial JSON means parsing an incomplete JSON string as tokens arrive.

Two practical approaches:

• Partial-JSON parsers (partial-json on npm, or Anthropic/OpenAI SDK helpers) that tolerate an unterminated string or array and return the best-effort object so far. Good for progressively revealing fields as they complete.
• Field-order-aware rendering: design your schema so the fields your UI shows first are also emitted first by the model (put a short summary field before a long details array). This costs you a little prompt engineering but means the user sees something meaningful within the first second of streaming instead of a spinner until the whole object closes.

import { parse } from "partial-json";

let buffer = "";
for await (const chunk of stream) {
  buffer += chunk;
  const partial = parse(buffer); // returns whatever is parseable so far
  updateUI(partial); // re-render with each new chunk
}

Don't try to validate partial JSON against your full schema mid-stream — validate only once the stream signals completion (a stop_reason: "end_turn" / finish event). Partial objects will always look "invalid" against a schema that requires fields not yet streamed.

Provider-specific notes (2026)

• Anthropic (Claude): the most reliable path is tool-use with a forced tool choice, not the plain "respond in JSON" instruction. Claude Opus 4.5 and Sonnet 4.5 both support strict input_schema validation server-side, rejecting malformed tool calls before they reach you. See Claude Opus 4 in depth for model-specific behavior.
• OpenAI: response_format: {type: "json_schema", strict: true} on GPT-5-class models genuinely constrains sampling — this is the closest thing to a hard guarantee currently available. Non-strict mode is best-effort only.
• Google Gemini: response_schema works well but historically has been pickier about additionalProperties: false — test explicitly rather than assuming parity with OpenAI's implementation.
• Local/open models via Ollama: grammar-constrained decoding (GBNF, outlines) exists but adds real latency overhead on consumer hardware; budget for it if you're going this route (see running local LLMs with Ollama).

If your app talks to more than one of these providers, don't hand-roll a schema adapter per vendor — a gateway layer that normalizes the request/response shape saves a lot of glue code; see Vercel AI Gateway for multi-model routing.

Where structured outputs fit in a larger system

Structured outputs are rarely the whole feature — they're the seam between an LLM and the rest of your stack. A few places this shows up:

• Tool-calling agents: every tool call is a structured output under the hood; get comfortable with schemas here before building multi-agent systems.
• RAG pipelines: when you ask a model to extract structured facts from retrieved documents rather than free-text answers, the same validate-and-retry discipline applies — see RAG for SMBs.
• Evals: once you have a schema, you have a natural unit of testing — assert on parsed fields, not on prose similarity. Read eval-driven AI development before you ship this to production.
• Guardrails: schema validation is your first guardrail layer, not your only one — pair it with the broader practices in LLM guardrails and safety.

Common mistakes

• Trusting "JSON mode" alone and skipping schema validation entirely — syntax-valid is not the same as structurally correct.
• Writing a schema with vague field names and no descriptions, then blaming the model for guessing wrong.
• Retrying identically on failure instead of feeding the validator's error back as context.
• Validating a partial stream against the full schema and treating every intermediate state as an error.
• Returning silent defaults on failure instead of surfacing and logging the raw bad output.

FAQ

What's the difference between JSON mode and function calling for structured outputs?

JSON mode constrains the output to be syntactically valid JSON but doesn't guarantee your specific keys or types are present. Function/tool calling asks the model to populate a named tool's input schema, which providers tend to enforce more strictly server-side. For anything beyond a toy project, prefer tool calling.

Do I still need to validate output if the API guarantees schema compliance?

Yes. "Strict" schema modes reduce failures dramatically but don't eliminate edge cases like a model returning a technically-valid but semantically wrong value (a negative price, a date in the wrong century). Validate business rules with your own schema library regardless of what the API promises.

How many retries should I allow before giving up?

Two to three is the practical ceiling. If a schema-constrained call with a clear error message fed back still fails after 2-3 attempts, the input itself is likely the problem (a corrupted PDF, an ambiguous document) rather than something more retries will fix.

Can I stream structured output and still validate it?

Only validate once the stream is complete. Use a partial-JSON parser to render progressively as tokens arrive, but run full schema validation only after the model signals the response is finished — partial objects will fail validation against required fields that haven't streamed yet.

Which model is best for structured extraction tasks?

As of mid-2026, Claude Opus 4.5 and GPT-5-class models both perform well on schema-constrained extraction, with Claude's tool-use path being especially robust for complex nested schemas. Test on your actual documents rather than trusting a general benchmark — extraction accuracy varies a lot by domain (invoices vs. medical records vs. contracts).

Is structured output slower or more expensive than free-text generation?

Schema-constrained decoding adds a small latency overhead (typically under 10%) because the sampler has to check the grammar at each token. It's a worthwhile trade: the retry loops you'd otherwise need for malformed free-text responses cost far more in both latency and tokens than the constraint overhead itself.

Get help wiring this into your product

If you're building an extraction pipeline, a form-filling agent, or anything that needs an LLM to talk to your database reliably, we can scope it in a free 30-minute call. Reach out through the contact form or message us directly on wa.me/972585802298.

Sources

• Anthropic — Tool use documentation
• OpenAI — Structured Outputs guide
• Google — Gemini API structured output
• JSON Schema specification
• Zod documentation