A Pydantic AI agent can return a perfectly valid object with the wrong answer inside it. An extracted invoice total can be a valid decimal while referring to the wrong line. A support classification can belong to the allowed enum while routing a customer to the wrong team.
Pydantic AI monitoring should separate output validation, execution reliability, and business correctness. Record the public request and typed result, inspect retries and tools when something goes wrong, and connect the result to an outcome your application can verify.
Agnost AI is an agent analytics and observability platform. Its Pydantic AI integration offers a direct interaction wrapper and an OpenTelemetry path. The wrapper below captures the final boundary. It does not automatically record every validation attempt.
Where Logfire, OpenTelemetry, and Agnost fit
Pydantic AI’s observability documentation describes its Logfire integration and support for other OpenTelemetry destinations. Use execution instrumentation when you need to inspect model calls, tools, and internal behavior. The Agnost interaction wrapper shown below records a different boundary: the sanitized request and final output.
Choose the boundary from the question. A slow run needs execution timings; a valid but incorrect extraction needs source evidence and reconciliation; repeated corrections need stable job or conversation identity. Follow Agnost’s integration setup for deeper tracing, and inspect one received trace before relying on retry or token fields. Keep unchecked results distinct from confirmed correct results.
Worked example: the subtotal passes validation
Imagine an invoice with a subtotal, tax, and amount due. A typed extraction agent returns a valid currency and decimal total, but it selects the subtotal. This example illustrates a failure mode; it is not a measured result from the code above.
| Evidence | Illustrative value | What it establishes |
|---|---|---|
| Source subtotal | 100.00 | One valid number on the document |
| Source tax | 20.00 | Another relevant field |
| Source amount due | 120.00 | The intended extraction target |
| Validated output total | 100.00 | Correct type, wrong meaning |
| Reconciliation result | Mismatch | Independent evidence of the error |
A type check cannot distinguish these numbers by itself. A domain validator can reject inconsistent totals if it has the relevant evidence and business rules. A post-processing reconciliation can catch a mismatch later. Record which check found the problem; otherwise both may appear as an undifferentiated agent error.
Keep the original output and later correction distinguishable. Overwriting the first value with the corrected one would make production quality look better than the user experienced.
In Agnost, review the sanitized interaction and available events by schema and agent version. If the document is sensitive, store an approved reference and the minimal diagnostic fields instead of attaching the full invoice to every record.
Metrics that answer useful Pydantic AI questions
Use these definitions to design your measurement plan. Calculate them from the required application or tracing evidence; they are not all precomputed dashboard metrics.
| Metric | Definition | Interpretation |
|---|---|---|
| Terminal validation failure | Runs ending in validation failure / eligible runs | Does not count internal retries that recovered. |
| First-pass validity | Outputs accepted without a validation retry / observed outputs | Requires retry evidence; do not infer it from terminal success. |
| Reconciliation mismatch | Outputs failing an independent check / outputs actually checked | Keep unchecked output separate from correct output. |
Three different meanings of success
| Signal | What it establishes | What it does not establish |
|---|---|---|
| Invocation returned | The call completed normally | The answer is accurate |
| Output passed validation | The result satisfied configured constraints | The source information is correct |
| Downstream operation confirmed | An external system accepted or completed work | Every statement in the answer is true |
Define these independently. If an extraction agent produces a schema-valid purchase order, record that as a valid extraction result. Record human correction or successful reconciliation later, when that evidence exists. This distinction keeps “100% valid JSON” from becoming “100% correct automation.”
Pydantic AI supports typed outputs and validation mechanisms; see its output documentation. Your application’s checks determine which business constraints are enforced.
Capture an existing typed agent
This example expects your existing agent to return a Pydantic model in result.output. It exports only fields you explicitly allow. Install agnost alongside your existing Pydantic AI dependencies and set AGNOST_ORG_ID in the server environment.
Example scope: the code below is an integration adapter for an existing application. Configure its framework, model, credentials, and output contract as described; the demo screenshots were not generated by running these snippets.
import os
import agnost
agnost.init(os.environ["AGNOST_ORG_ID"])
def extract_with_tracking(agent, prompt, *, user_id, conversation_id):
interaction = agnost.begin(
user_id=user_id,
conversation_id=conversation_id,
agent_name="pydantic-invoice-extraction",
input="Extract approved invoice fields", # No raw document content.
properties={
"framework": "pydantic-ai",
"schema_version": "invoice-v1",
"agent_version": "extraction-v1",
},
)
try:
result = agent.run_sync(prompt)
safe_output = result.output.model_dump_json(
include={"currency", "total"},
)
except Exception as exc:
interaction.end(output=type(exc).__name__, success=False)
raise
else:
interaction.end(output=safe_output, success=True)
return resultAdapt the allowlist to your actual schema. A string-output agent needs a string serializer instead. Avoid serializing dependency objects or all message history simply because they are available. The example intentionally omits the original invoice; include sanitized evidence only if your data policy permits it.
In an asynchronous server, use await agent.run(...) in an async handler rather than calling run_sync() inside its event loop. Initialize Agnost once and call agnost.shutdown() on process exit. The success flag above represents normal execution, including successful serialization—not semantic correctness.
Expose retries without confusing them with demand
A single user request can produce several model attempts before it satisfies validation. Counting each attempt as another user interaction inflates usage and obscures reliability.
Keep the user-turn record at the application boundary. Use traces for internal attempts. When latency rises, determine whether it came from model response time, tool execution, or repeated correction after validation failed. A lower terminal error rate may be accompanied by more retries and slower responses.
Use the documented Logfire/OpenTelemetry path for deeper instrumentation. Pydantic AI describes both Logfire and alternative OpenTelemetry collection in its observability guide. Configure the destination and content recording intentionally, then inspect the emitted spans. Do not assume every exception message or validation payload is appropriate to export.
Diagnose valid but wrong output
For an invoice example, investigate the evidence in this order:
- Did the agent receive the correct document and page?
- Did OCR or a tool preserve the relevant total and currency?
- Did the output validator enforce only type, or also domain constraints?
- Was a later correction needed?
A number can pass type validation while being the subtotal instead of the amount due. Store an approved source reference or field provenance when your application has it. That makes corrections explainable without retaining the entire document in every span.
If a validator rejects output, record a stable reason category such as missing required field or unsupported currency. Avoid using unbounded raw error strings as metric dimensions. Group by schema and agent version so a rollout that tightens validation is not mistaken for a sudden decline in model quality.
Review conversations and downstream results together
In Agnost Users, inspect related interactions; in Events, inspect the records received. An extraction pipeline may not resemble a chat, but a stable conversation or job grouping still lets you connect submission, retry, and correction.
Useful review questions include: which document types need repeated attempts; which field is most often corrected; whether a schema revision moved errors upstream; and whether successful extraction actually led to a completed workflow. These outcomes require evidence from your application. Agnost cannot infer an unrecorded reconciliation result from a well-formed object.
Frequently asked questions
Why are my traces missing the output?
Check whether content recording is enabled, whether your output uses the expected schema, and whether your exporter sends the relevant spans. A request wrapper can also intentionally omit fields through its allowlist.
Does validation remove the need for monitoring?
No. Validation enforces the checks you define. Production monitoring reveals missing checks, changing inputs, tool failures, and valid outputs that users still correct.
How should I compare two agent versions?
Compare the same input categories and schema contract. Report validation failures, retry behavior, and independently observed corrections separately. A version that accepts more outputs is not necessarily more accurate.
Start with Agnost’s Pydantic AI setup. For programs whose prompts are optimized rather than authored directly, see DSPy monitoring.
Browse all agent observability guides by framework.