Schema validation proves that an output has an allowed structure. It does not prove that its values are true, authorized, or supported by an executed action. An agent can return perfectly valid JSON for the wrong customer, an impossible date, or a refund that never happened.
Use structural validation as the first check, then apply the business contract for the operation.
Validate in layers
| Layer | Example check | What can still go wrong? |
|---|---|---|
| Parsing | The payload is valid JSON | Wrong fields or types |
| Schema | Required fields, enums, types, allowed keys | Plausible but false values |
| Domain | Amount, currency, date, and state rules | Wrong customer or object |
| Authorization | Caller can act on this object | Operation still may not execute |
| Outcome | Matching authoritative result confirms the state | User-facing wording may still exaggerate it |
The JSON Schema object reference explains properties, required fields, and additional-property constraints. Those mechanisms describe data shape. They are not a substitute for checking the data against your application.
A valid payload can make a false claim
Consider this illustrative structured result:
{
"order_id": "order_42",
"refund_status": "completed",
"amount_minor": 2500,
"currency": "USD"
}It may satisfy a schema with string IDs, a status enum, and a nonnegative integer amount. But the actual request might concern order_99, or the refund may still be pending. To validate the result, compare it with the authenticated request and an authoritative action record.
Never use a model-generated authorized: true field as the authorization decision. Authorization belongs at the execution boundary, where the system knows the caller and resource.
Keep generation and execution contracts separate
It is often useful for a model to propose an action in structured form. Treat that object as a proposal. Validate it, authorize it, execute through trusted code, and then construct the result from what happened.
This gives you two different records: requested action and execution outcome. Reusing one model-authored object for both makes it difficult to tell whether “completed” means the model wanted to complete the task or the system actually did.
Use stable object references and preserve the relationship between the proposal and its result. If the tool resolves a human-readable name to an internal ID, record the resolved identity so a reviewer can investigate a wrong-object action.
Design failures that your schema will not catch
Create a test with an existing but unauthorized object, a valid currency paired with the wrong amount, a status inconsistent with the provider response, and an ISO-formatted date outside the permitted interval. Include a correct output as a control.
Then verify which layer rejects each case. The result should be a specific reason, not a generic “invalid output” that encourages the agent to keep trying different values until something passes.
If the agent repairs malformed output, bound the repair attempts and preserve the original error. A successful JSON repair may still leave the semantic error unchanged. The retry-budget guide describes how to prevent those repair loops from consuming an entire task.
Measure structure and meaning separately
Track parse failures, schema failures, domain rejections, authorization rejections, and unsupported outcome claims as distinct categories. A schema improvement may reduce parsing errors while the rate of wrong-object actions stays flat.
For model-based semantic review, use a labeled set and allow unknown results when the reference evidence is missing. A judge that sees only the output cannot verify account ownership or payment state reliably.
Agnost can connect the tool execution record to the conversation that produced it. Supply the relevant approved result fields so a finding can be investigated without guessing at unseen application state.
Use the false-success detector for completed-action claims and the regression-test workflow to keep valid-but-wrong examples in your test suite. A well-formed response is useful only when it represents the right result.