A practical guide to structured LLM outputs

Structured output is not just asking AI to return JSON. It means defining a contract between the model and the application, validating that contract, and deciding what happens when the output does not match the expected format.

Context

In the previous guides in this series, the goal was to build a practical foundation for thinking about AI applications.

First, the focus was RAG: when an application needs to retrieve external context before answering. Then the focus was chunking, embeddings, and retrieval: how to prepare and retrieve context before giving it to the model.

But another problem appears quickly when AI stops being only a conversation and starts entering a real application: the model response needs to become data.

It may need to fill fields, feed a screen, pass through a rule, generate a suggested action, classify a message, summarize a request, create a list of next steps, or prepare something for human review.

In that scenario, a good-looking answer is not enough. If the system expects an object with specific fields, defined types, and allowed values, a well-written paragraph can be useless. Worse: it may look correct to a person while breaking the application flow.

The problem

The central question of this guide is: how can you use LLM responses inside an application without depending on free text and manual interpretation?

Asking the model to "respond in JSON" may seem enough at first. In a simple prototype, it may even work. But when the output enters a system flow, more concrete questions appear.

• Which fields are required?
• Which types are accepted?
• Which values are valid?
• What happens if a field is missing?
• Does the system retry, show an error, or send it for review?
• Does validation check only the format or also the quality of the content?

Without answering these questions, the integration becomes fragile. The model generates something, the code tries to interpret it, and the application hopes the output matches what was expected.

Mental model

1expected contract2  -> response generation3  -> validation4  -> system usage or fallback

The expected contract defines the output format. Generation guides the model to answer in that format. Validation checks whether the received response can be used with minimum safety. Fallback defines what happens when the output does not pass validation.

Structured output is not a guarantee of correctness. It is a way to make explicit what the system expected to receive.

Free text, JSON, and schema

Free text is when the model responds in natural language, without a rigid format. JSON is a data format that code can read. Schema is the definition of the expected format: which fields exist, which are required, which types are accepted, and which limits need to be respected.

A JSON object can be well formatted and still be bad for the application.

1{2  "priority": "urgent",3  "summary": "",4  "suggested_actions": []5}

If the system only accepts `low`, `medium`, or `high` as priority, the value `urgent` is outside the contract. If `summary` is required and cannot be empty, that field failed. If `suggested_actions` needs at least one item, the empty list also needs to be handled.

That is why the main point is not using JSON. The main point is defining a contract the system can validate.

When structured outputs help

Structured outputs are useful when the model output needs to be consumed by code.

• Extract data from a message.
• Classify a request.
• Fill form fields.
• Generate a task list.
• Identify user intent.
• Build parameters for a search.
• Suggest actions for human review.
• Return blocks that the interface will render.

Practical example

Imagine a fictional application that receives support messages and uses AI to organize triage.

1I cannot access my account since yesterday. I already tried to reset the password, but the email does not arrive.

A free-text answer could be readable for a person, but not very practical for the system. A structured output can organize the same case into useful fields.

1{2  "type": "support",3  "category": "login",4  "priority": "medium",5  "summary": "The user cannot access the account and does not receive the password reset email.",6  "suggested_actions": [7    "Check password reset email delivery logs",8    "Confirm whether the registered address is correct",9    "Check recent login attempts"10  ],11  "needs_human_review": true12}

Now the application can filter by category, highlight priority, fill a screen, send the case for review, store the summary, and show suggested actions without interpreting a paragraph.

When free text still makes sense

Not every answer needs to be structured. Free text can be enough when the goal is to talk, explore ideas, explain a concept, write a narrative draft, summarize something for human reading, generate copy variations, or support a decision without triggering an automated flow.

The criterion is simple: does the application need to use this response as data? If yes, structure helps. If not, free text may be enough.

How to think about schema

A good schema starts from the application need, not from the answer that looks nicest. Before asking the model for any format, it is worth asking which fields the system really needs, which values are allowed, which fields can be optional, and what will be shown to the user.

1type: support | question | feedback | other2category: login | billing | account | product | other3priority: low | medium | high4summary: required short text5suggested_actions: list of text items6needs_human_review: boolean

This contract does not need to be complex to be useful. It only needs to make clear what the application expects.

Validation: format is not quality

A common trap is assuming that if the response passed parsing, it is good. Parsing, format validation, and quality validation are different layers.

• Parsing: could the system read the output?
• Format validation: do fields and types respect the contract?
• Quality validation: does the content make sense, is it correct, and is it useful for the case?

A response can be valid JSON, pass the schema, and still contain a poor classification. Structured outputs help integration, but they do not replace evals, human review, and quality criteria.

Common mistakes

Asking for JSON without defining a contract

"Respond in JSON" is an instruction, not a contract. If the prompt does not define fields, types, accepted values, and limits, the model can invent a different structure for each answer.

Accepting the response without validation

Even if the model usually gets the format right, the application should not assume the output will always be correct. Fields may be missing, types may change, lists may be empty, and values may fall outside the expected set.

Confusing structure with truth

Schema helps validate format; it does not guarantee that the model understood the case, used the right context, or classified it well. Even if the model returns perfect JSON, it may have used weak or insufficient retrieved context.

Hiding failures from the user

When the response does not pass validation, the application needs to handle it clearly. A parsing failure should not become a broken screen, a wrong automated action, or a confusing message.

Fallback: what to do when it fails

Fallback is the part many people leave for later, but it should be part of the design from the beginning.

• Retry with a stricter instruction.
• Return a controlled error.
• Ask for human review.
• Use a manual path.
• Save the response as pending.
• Log the case for analysis.

The best fallback depends on risk. If the output only suggests tags for an internal draft, a retry may be enough. If the output influences a sensitive action, the path needs to be more conservative.

A practical flow example

11. receive the user message22. build the prompt with the expected contract33. request structured output44. try to parse it55. validate required fields, types, and accepted values66. if it passes, use the data in the system77. if it fails, apply fallback88. log the result for review

Minimum validations can check whether `type` and `category` are in the allowed list, whether `priority` is `low`, `medium`, or `high`, whether `summary` is not empty, whether `suggested_actions` is a list, and whether `needs_human_review` is boolean.

This validation still does not say whether the triage is perfect. But it already prevents malformed output from entering the rest of the system as if everything were fine.

How to validate whether it is working

A good start is to create a small set of known cases. For each case, record the original input, expected output, received output, format error, content error, applied fallback, and human decision when available.

1Input: "I do not receive the password reset email."2Expected: login category, medium priority, human review true3Received: account category, low priority, human review false4Diagnosis: valid format, weak classification5Next adjustment: improve examples and priority criteria

This record separates two types of problems: the response could not be read or did not respect the schema; or the response respected the format but was not good. This separation prepares the ground for evals and harnesses.

When to keep it simple

Structured outputs are useful, but they do not need to become over-engineering. You may not need a large schema if the response only supports human reading, if the flow is still exploratory, if the task does not trigger anything in the system, or if manual review always happens before use.

1{2  "summary": "short text",3  "needs_review": true4}

The goal is not to make everything more complex. It is to make the minimum contract explicit when the application needs to rely on some form of data.

Lessons learned

• Structured output is useful when AI output needs to be consumed by code.
• JSON is only a format; the main point is the output contract.
• Schema helps make fields, types, requirements, and limits explicit.
• Parsing, format validation, and quality validation are different things.
• A structured response can be well formatted and still be wrong.
• Fallback needs to be designed before failure happens.
• Human review remains important in sensitive flows.
• Logging failures helps improve prompts, schemas, evals, and harnesses.

Limits and caveats

This guide does not try to choose a specific API, SDK, or framework. It also does not go into advanced data validation, schema versioning, or provider comparisons.

Another important limit: structured output does not solve factual problems. If the retrieved context is poor, if the prompt is confusing, or if quality criteria do not exist, the model can still return poor output inside a valid format.

Conclusion

Asking an LLM to respond in JSON can be a start, but it is not enough. In real applications, the broader question is: what contract does the application expect to receive?

If the response needs to become data, the output should be predictable, validatable, and handled when it fails. This does not eliminate errors, but it removes part of the integration from the realm of luck.

Before using AI output in the system, define the expected format, validate what arrived, and have a clear path when the model does not fulfill the contract.