The Output Contract: Validating LLM JSON Before Production
Parsing a model response is not the same as validating it. This guide separates syntax, schema and semantic checks, maps each to official sources, lists the failure modes that reach production, and gives a go-live checklist you can hand to a reviewer. Include…
Halaman ini belum dilokalkan — materi sumber ditampilkan dalam bahasa aslinya.
Why an output contract belongs in the deployment, not the prompt
On a customer deployment, a language model is rarely the last component in the chain. Its output is parsed, written into a database, forwarded to a ticketing system, or shown to an operator who has to act on it. The interesting failure is not the model being wrong in an obvious way. It is the model returning something that looks perfectly plausible, parses as JSON, and still has a missing field, a number where the downstream service expects a string, or an extra key that quietly changes a branching decision. Everything upstream looks healthy. The incident appears three systems later, in someone else’s queue.
Inference. This is why we treat the output contract as a deployment boundary rather than a prompt-engineering detail. A prompt asks nicely. A contract decides whether the request is allowed to continue. Forward-deployed engineers are usually the ones who have to make that distinction concrete before go-live, because they are accountable for what happens on the customer’s production system, not for how elegant the demo looked.
What the standards actually say
Fact. JSON Schema is the standard vocabulary for describing the shape of a JSON document. Its keyword set includes type, required and additionalProperties, which let a consumer state which fields must exist, what type they must have, and whether unknown keys are permitted [S1]. The Model Context Protocol specification uses JSON Schema for tool input and output schemas, so the same vocabulary governs what a tool declares it accepts and returns [S3]. Together these two official sources establish that schema validation is a defined, portable contract mechanism and not a vendor convention.
Fact. Python’s standard json module exposes json.loads for decoding a JSON document, and malformed input raises JSONDecodeError [S2]. LangChain’s official documentation documents model integration and structured output as first-class concerns in an application stack [S4]. We cite these four sources for definitions and API behaviour only; we do not reproduce their text, and we do not claim any performance characteristic from them.
Inference. Note what those facts do not give you. A schema tells you the shape is legal. It does not tell you the content is true, that the value is the right one for this customer, or that the downstream system will accept it. A parser tells you the bytes are JSON. It does not tell you a required field survived. These are three separate checks, and collapsing them into one “validate the response” step is the most common reason a validation layer fails in production.
Three layers, in order
Layer 1 — Syntax. Parse the raw response with a real parser and treat a parse error as a first-class, observable outcome. Do not regex your way to a dictionary, and do not silently strip prose around a JSON block. If the model emits text before the object, that is a contract violation with a cause you can investigate, not a cleaning step to hide [S2].
Layer 2 — Shape. Validate the parsed object against a versioned schema: required fields present, types correct, enums respected, unknown keys either rejected or explicitly allowed. Keep the schema next to the code that consumes the object, and version it in the same repository. additionalProperties: false is a deliberate decision, not a default: rejecting unknown keys is safer for a stable downstream contract, while allowing them is kinder to a model that is still being tuned [S1][S3].
Layer 3 — Meaning. Assert what the schema cannot express. A confidence value of 0.0 may be perfectly valid JSON and still be useless. A date may be a string and still be in the past when it must be in the future. A ticket category may be in the enum and still contradict the ticket text. Write these as named assertions with named failure reasons. They are business rules, they change more often than the schema, and they deserve their own test suite.
Advice. Fail closed at layer 1 and layer 2, but decide deliberately at layer 3. A hard failure is right when a wrong value would write bad data; a fallback or human review is right when the cost of a refusal exceeds the cost of a check. Record which branch you took. “The model returned something odd” is not an incident report; “layer 2 rejected missing field customer_id, request routed to retry, then to review” is.
Illustrative example (not measured)
Example, not a benchmark. The sketch below shows the shape of a three-layer gate. It is illustrative scaffolding: it has not been benchmarked, it is not a library, and it makes no accuracy, latency or cost claim. Adapt it to your own schema and your own failure policy before using it in production.
import json
from json.decoder import JSONDecodeError
def accept(raw, schema_validate, semantic_checks):
# Layer 1: syntax. Never strip or "repair" silently.
try:
obj = json.loads(raw)
except JSONDecodeError as exc:
return reject("syntax", str(exc))
# Layer 2: shape, against a versioned schema.
errors = schema_validate(obj)
if errors:
return reject("shape", "; ".join(errors))
# Layer 3: business meaning, named assertions.
for name, check in semantic_checks.items():
if not check(obj):
return reject("semantics", name)
return accept_object(obj)
def reject(layer, reason):
# Log the layer and reason; do not log the raw payload if it may carry PII.
return {"ok": False, "layer": layer, "reason": reason}
Advice. Log the failing layer and reason, not the raw payload: on a real deployment the payload may contain customer data. A validation log that leaks the very data you were trying to protect is a worse outcome than the malformed response.
Failure modes worth writing down before go-live
| Failure | Which layer catches it | What to record |
|---|---|---|
| Prose wrapped around the JSON object | Syntax | Parse error, model/version, prompt template id |
| Required key missing or null | Shape | Schema version, field name |
| Number where a string was declared | Shape | Schema version, declared vs observed type |
| Extra key silently added | Shape (if additionalProperties: false) | Key name, decision to reject or allow |
| Valid JSON, wrong business value | Meaning | Assertion name, input features, routing decision |
| Retry storm after a validation failure | Policy | Retry count, backoff, circuit-breaker state |
Inference. The last row is the one teams forget. Once you start rejecting outputs automatically, a systematic model or prompt regression can turn into a retry storm. A bounded retry with backoff, a circuit breaker, and a path to human review are part of the contract, even though no schema expresses them.
A go-live checklist you can actually hand to a reviewer
- Name the downstream consumer and the exact fields it requires, with an owner for each.
- Keep one versioned schema in the repository; reference it from code, never paste a second copy into a prompt.
- Decide
additionalPropertiesexplicitly and write down why. - List the semantic assertions separately from the schema, with named failure reasons.
- Log the failing layer and reason; never log raw payloads that may contain personal data.
- Bound retries, add backoff, and define what happens when the breaker opens.
- Add a regression fixture for every real validation failure you see in production.
- Record the model and prompt-template version alongside each validation outcome.
Advice. Bring the checklist to the go-live review with the failure log from a staging run. A reviewer can accept a contract they can see failing in a controlled way; they cannot accept a verbal assurance that “the model usually returns JSON”.
Limits of this guide
We did not run an accuracy, latency or cost benchmark for anything described here. No framework is claimed to be better than another, no schema library is endorsed, and the code sketch is an illustrative example rather than tested production code. The official sources below are cited for definitions and API behaviour only; their text is not reproduced, and no commercial reuse licence is asserted. Version-sensitive details such as API parameters, model capabilities and pricing must be re-checked against the linked official documentation on the date you deploy — this guide records a verification date, not a permanent guarantee.
Sources
- [S1] JSON Schema Specification — standard_spec; title and canonical link only, no full-text reproduction; licence UNDECLARED / index_only; verified 2026-11-07.
- [S2] Python Standard Library · json — official_api_docs; title and canonical link only, no full-text reproduction; licence UNDECLARED / index_only; verified 2026-11-07.
- [S3] Model Context Protocol · Server Tools (spec 2025-06-18) — standard_spec; title and canonical link only, no full-text reproduction; licence UNDECLARED / index_only; verified 2026-11-07.
- [S4] LangChain OSS Python Documentation — official_product_docs; title and canonical link only, no full-text reproduction; licence UNDECLARED / index_only; verified 2026-11-07.
Statement types used above: Fact = traceable to an official source; Inference = our engineering reasoning; Advice = a practice recommendation; Example = illustrative code, not a measurement.
Yang bisa Anda lakukan sekarang
- Parsing, shape validation and semantic assertions are three different checks
- Version the schema next to the code that consumes the object
- Log the failing layer and reason, never the raw payload
- Bound retries: automatic rejection can become a retry storm
FDE Deployment Toolkit 2026
Arsitektur referensi, checklist go-live 60 poin, model biaya GPU, dan daftar pendek vendor — empat artefak yang selalu Anda bangun ulang di setiap deployment.