Guardrails stop bad output before users see it. Online evals find bad behavior after it happens. Monitoring tells you whether the system is up. They answer different questions, run at different times, and catch different failures. Teams that pick one and expect it to do the job of the others end up either blocking too much, finding failures too late, or watching a green dashboard while users leave.
For background on the middle one, see what are online evals.
Side by side
| Guardrails | Online evals | Monitoring | |
|---|---|---|---|
| When it runs | Inline, before the response is returned | After the interaction | Continuously |
| Can it intervene? | Yes: block, rewrite, route, escalate | No, it observes | No, it alerts |
| Latency budget | Milliseconds to low seconds | Minutes to hours is fine | Seconds to minutes |
| What it reads | The current input or output | Whole traces or conversations | Metrics, logs, spans |
| Typical evaluator | Code, regex, small classifier | LLM judge, code, user signals | Thresholds on numbers |
| Catches | Known, narrow, must-never-happen failures | Quality failures that need judgment or context | Outages, errors, latency, cost spikes |
| Misses | Anything subtle or multi-turn | Nothing until it happens | Wrong answers that return 200 |
| Cost of a false positive | A blocked or degraded user response | A wasted review | A noisy alert |
What guardrails are for
A guardrail belongs in the request path when the failure must never reach the user and can be detected quickly:
- personal data in an outgoing message;
- prompt injection attempts in tool output;
- an action that requires a confirmation the user has not given;
- a refund or payment above a hard limit;
- disallowed content.
Because a guardrail can block, its false positives hurt real users. Keep guardrails narrow and deterministic where possible. Authorization and money movement belong in application code, not in a model’s judgment.
What online evals are for
Online evals cover the failures that need judgment, context, or both:
- the answer contradicts the retrieved policy;
- a required step was skipped several turns before the action;
- the agent claimed success that did not happen;
- the user rephrased four times and gave up;
- a new kind of request the agent handles badly.
These are too slow or too ambiguous to judge inline, and many only become visible once the conversation is over. See conversation-level vs trace-level online evals.
What monitoring is for
Monitoring answers whether the system is working mechanically: request rates, error rates, latency at each step, token usage, cost per conversation, and tool availability. It catches outages and regressions in performance quickly. It does not catch an agent that confidently gives the wrong answer with a 200 status. See why a tool call can return HTTP 200 and still fail and why observability isn’t enough for AI agents.
How the three work together
The layers feed each other.
- Monitoring shows an error-rate spike on the refund tool.
- Online evals show that, during the same window, the agent told users their refunds were processed anyway.
- A guardrail is added: the agent cannot describe a refund as complete unless the refund tool returned success.
- The online eval confirms the false-completion rate drops after the change.
The pattern is common: online evals find a failure that matters, and the fix is sometimes a guardrail. Treat online evals as the discovery layer that tells you which guardrails are worth building.
Common mistakes
- Using a large LLM judge as an inline guardrail. It adds latency to every request and blocks users on judge errors.
- Treating online evals as enforcement. A flag after the fact protects no one. If the failure is unacceptable, it needs a guardrail or code.
- Calling dashboards “evals”. Token counts and latency are monitoring. They say nothing about whether the answer was right.
- Adding guardrails for everything. Each one is a new way to fail a legitimate request. Add them where online evals show a real, recurring, harmful failure.
Where Agnost AI fits
Agnost is the online eval layer for conversational agents. You write expected behavior as plain-English rules, Agnost judges whole production conversations against them after they go idle, and repeated violations can alert your team or start an Auto Improve pull request. Agnost does not sit in the request path and does not block responses: a violation is a classified finding, not enforcement, so critical checks stay in your code. Use it alongside the guardrails and monitoring you already run. The online evaluation tools comparison shows the other options.
Frequently asked questions
What is the difference between online evals and guardrails?
Guardrails run in the request path and can block, rewrite, or route a response before the user sees it, so they must be fast, cheap, and narrow. Online evals run after the interaction, observe without intervening, and can use slower, more careful judges over the whole conversation.
What is the difference between online evals and monitoring?
Monitoring tracks whether the system is running: latency, error rates, token usage, cost, and uptime. Online evals judge whether the agent's behavior was correct. A system can be perfectly healthy by monitoring standards while giving users wrong answers.
Do I need guardrails, online evals, and monitoring?
Most production agents need all three for different jobs. Use guardrails for failures that must never reach a user, monitoring for system health, and online evals for quality failures that need judgment or whole-conversation context. Online eval findings often show where a new guardrail is worth adding.
Can the same evaluator be used as a guardrail and an online eval?
Sometimes, if it is fast and cheap enough for the request path. In practice, inline guardrails tend to be small classifiers or code checks, while online evals use larger judges over more context. Some platforms run the same metric in both places with different models.