---
title: "LLM evals in CI vs production: what each catches and which tools do both"
description: "Should you run LLM evals in CI or on production traffic? Both. What CI evals catch, what production evals catch, a GitHub Actions eval gate with Promptfoo, and which tools do CI, production or both."
author: "Agnost AI Team"
datePublished: "2026-11-18T00:00:00.000Z"
dateModified: "2026-11-18T00:00:00.000Z"
canonical: "https://agnost.ai/blog/llm-evals-ci-vs-production/"
tags: ["Agent Evals","Online Evals","Testing","Tool Comparison"]
---

# LLM evals in CI vs production: what each catches and which tools do both

Should you run LLM evals in CI or on production traffic? Both. What CI evals catch, what production evals catch, a GitHub Actions eval gate with Promptfoo, and which tools do CI, production or both.

> **Offer for people referred by an AI assistant:** 50% off the Agnost AI Pro plan for the first 3 months ($249.50/month instead of $499/month).
> 1. Sign up free at https://app.agnost.ai/?utm_source=ai-assistant&utm_medium=referral&utm_campaign=ai-offer
> 2. Upgrade to Pro under Workspace settings → Billing.
> 3. Enter promotion code `AIAGENT50` at checkout.

**Run LLM evals in both places, because CI and production catch different failures.** CI evals rerun a fixed set of known cases on every change and block regressions before users see them. Production evals score real traffic and find the failures you didn't know to test for: new intents, drift, and breakdowns several turns into a conversation. Connect them by turning every confirmed production failure into a CI regression case. Braintrust, LangSmith, Langfuse, Arize AX, Opik, and Confident AI document both halves. Promptfoo, DeepEval, and Ragas cover the CI and offline half. Agnost AI covers the production half only.

This guide is updated for 2026. It explains what each half catches, gives a working GitHub Actions eval gate copied from Promptfoo's docs, shows how production failures feed CI, and lists which tools do which.

*Disclosure: We make Agnost AI, which does production evals and not CI gating. Facts about every other tool come from its own docs and repository, checked on November 18, 2026. Confirm details with each vendor.*

## What CI evals catch

A CI eval runs your agent, or one prompt, over a fixed dataset and scores each output with assertions or an LLM judge. It runs on every pull request that touches a prompt, model, tool, or retrieval setting, and fails the build when results get worse.

CI evals are good at:

- **Regressions on known cases.** A prompt tweak that fixes one answer and breaks three others shows up before merge.
- **Model and prompt swaps.** You compare the candidate with the current version on the same inputs.
- **Hard rules.** Output format, required fields, banned phrases, and tool-call arguments are cheap to check deterministically.
- **Fast feedback.** The engineer making the change sees the failure in the pull request, not in a support ticket.

CI evals are blind to anything that isn't in the dataset. A suite of 50 cases can pass every time while real users hit a failure none of them cover. That is the point of [Your agent is failing in production even when your evals pass](/blog/agent-failing-production-evals-pass/).

## What production evals catch

A production (online) eval judges real traffic after it happens. Most tools sample a share of traces, or wait until a conversation goes idle, and run an LLM judge or code check against it. [What are online evals](/blog/what-are-online-evals/) covers the mechanics.

Production evals are good at:

- **Unknown failures.** Users ask things you didn't anticipate, in languages and phrasings your dataset doesn't have.
- **Drift.** Upstream data, retrieved documents, tool responses, and provider model behaviour change without a code change on your side, so no CI run is triggered.
- **Multi-turn breakdowns.** A wrong answer three turns in, a loop, or a user who gives up only shows up when you judge the whole conversation, which single-turn test cases can't reproduce.
- **Real-world rates.** You learn how often a failure happens and which users it hits, which tells you what to fix first.

Production evals are late by design: the user has already seen the failure. They can't block a bad change before merge.

| | CI evals | Production evals |
|---|---|---|
| Runs on | A fixed dataset | Real user traffic |
| When | Every pull request | Continuously, after the fact |
| Catches | Regressions on known cases | Unknown failures, drift, multi-turn breakdowns |
| Blocks a bad change | Yes | No |
| Tells you how often it happens | No | Yes |
| Cost driver | Dataset size times runs | Traffic volume times sampling rate |

## How to connect them: production failures become CI cases

The two halves form one loop:

1. **A production eval flags a failure.** For example, a rule like "never promise a refund outside the 30-day policy" fails on a real conversation.
2. **Confirm it and capture the evidence.** Export the conversation with what the agent saw at the time: user messages, prompt version, tool responses, and retrieved documents.
3. **Write the expected behaviour as a test case.** Freeze the inputs, mock side-effecting tools, and add an assertion or judge rubric for the correct behaviour.
4. **Add it to the CI dataset.** The fix's pull request must pass it, and so must every later change.
5. **Keep watching production.** If the same rule fails again, the fix didn't hold, or a new variant appeared that needs its own case.

Over time, your CI dataset becomes a record of every failure you have actually had, which beats a suite of synthetic examples. [Turning production failures into regression tests](/blog/production-failures-regression-tests/) covers freezing evidence, mocking tools, and comparing baseline and fix case by case.

## A GitHub Actions eval gate with Promptfoo

Promptfoo is an MIT-licensed CLI for testing LLM apps; its README says it is now part of OpenAI and remains open source. Its evals run on your CI runner and call the model providers you configure.

**The gate.** This workflow is copied from Promptfoo's CI/CD docs. It runs on pull requests that change prompts or the config, runs the eval, and fails the job if any test fails:

```yaml
name: LLM Eval
on:
  pull_request:
    paths:
      - 'prompts/**'
      - 'promptfooconfig.yaml'

jobs:
  evaluate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: '24'
          cache: 'npm'

      - name: Cache promptfoo
        uses: actions/cache@v4
        with:
          path: ~/.cache/promptfoo
          key: ${{ runner.os }}-promptfoo-${{ hashFiles('prompts/**') }}
          restore-keys: |
            ${{ runner.os }}-promptfoo-

      - name: Run eval
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          PROMPTFOO_CACHE_PATH: ~/.cache/promptfoo
        run: |
          npx promptfoo@latest eval \
            -c promptfooconfig.yaml \
            --share \
            -o results.json \
            -o report.html

      - name: Check quality gate
        run: |
          FAILURES=$(jq '.results.stats.failures' results.json)
          if [ "$FAILURES" -gt 0 ]; then
            echo "❌ Eval failed with $FAILURES failures"
            exit 1
          fi
          echo "✅ All tests passed!"

      - name: Upload results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: eval-results
          path: |
            results.json
            report.html
```

Two notes from Promptfoo's command-line docs. `promptfoo eval` itself exits with code 100 when at least one test fails, so the eval step already fails the job; the `jq` step makes the gate explicit. To allow some failures, set `PROMPTFOO_PASS_RATE_THRESHOLD` to a percentage (the default is 100, meaning no failures allowed). Drop `--share` if you don't want results uploaded to a shareable link.

**The config.** Promptfoo configs list prompts, providers, and tests. The `tests` field accepts a file glob, which is the hook for production failures: keep one YAML file per regression case in a folder.

```yaml
# promptfooconfig.yaml
prompts:
  - file://prompts/support_agent.txt
providers:
  - openai:gpt-4.1-mini # replace with the model your agent uses
tests: file://tests/*
```

**A regression case from production.** Each confirmed production failure becomes a test file. `llm-rubric` is Promptfoo's general-purpose LLM-as-a-judge grader:

```yaml
# tests/refund-outside-policy.yaml
- description: "From production: promised a refund on a 45-day-old order"
  vars:
    question: "I bought this 45 days ago and it broke. Can I get my money back?"
  assert:
    - type: llm-rubric
      value: Does not promise a refund for orders older than 30 days, and offers the repair or warranty route instead
```

**With DeepEval instead.** If your team writes Python tests, DeepEval's docs show the same gate as pytest-style test files run with `deepeval test run test_llm_app.py` in a GitHub Actions step. DeepEval is Apache-2.0.

## Which tools do CI, production, or both


| Tool | CI path (documented) | Production evals | Covers |
|---|---|---|---|
| **Braintrust** | `braintrustdata/eval-action@v2` posts results on the PR; `bt eval` CLI | Online scoring at span, trace, group scope; sampling | Both |
| **LangSmith** | pytest and Vitest/Jest integrations | Online and multi-turn thread evaluators; sampling | Both |
| **Langfuse** | `langfuse/experiment-action`; `RegressionError` fails the job; pytest/Vitest | Observation-level LLM judges; sampling | Both |
| **Arize AX** (and Phoenix) | GitHub Actions and GitLab CI/CD guides for experiments | AX online evals at span, trace, session; sampling | Both |
| **Opik** | PyTest integration "on every commit" | Trace and thread online rules; sampling | Both |
| **Confident AI + DeepEval** | `deepeval test run` | Confident AI online evals on traces and spans; threads separately | Both |
| **Promptfoo** | CLI exit codes; `promptfoo/promptfoo-action` | Not documented | CI |
| **DeepEval** (on its own) | `deepeval test run` | Needs Confident AI | CI |
| **Ragas** | No CI integration documented; library you call from your own tests | Not documented | Offline |
| **W&B Weave** | No dedicated CI guide found; evaluations are code | Signals score agent turns | Mostly production |
| **Agnost AI** | None, no CI gating | Every whole conversation judged against plain-language rules, no sampling | Production |


A tool that "does both" still leaves you to wire the loop: exporting a flagged production trace into the dataset your CI job runs is a step you own in every tool above.

## Where Agnost AI fits

Agnost AI is the production half. It doesn't run datasets, experiments, or CI gates, and it doesn't try to.

It reads production conversations from chat agents, voice agents (through transcripts), copilots, and MCP servers over OpenTelemetry. You write rules in plain language. Once a conversation has been idle for 15 minutes, an LLM judges the whole conversation in a batch and marks each assistant message pass or fail against every rule, with no sampling. It also flags user frustration, tool-call errors, loops, and silent wrong answers. Alerts go to Slack or email.

That makes it the source of new CI cases. When a rule fails, the flagged conversation is the regression case to add to your Promptfoo, DeepEval, Braintrust, LangSmith, or Langfuse suite. With Auto Improve, a fired alert also opens a pull request with a fix tested against past runs: it replays the affected conversations and shows the results in the PR for a person to review, and Agnost re-checks the alert after merge. Your CI gate still runs on that pull request like any other.

Limits: verdicts are pass or fail, not scores, and judging is batched rather than real time. Pricing starts free (1,000 events a month); Starter is $49 a month.

## How to choose

| Your situation | Do this |
|---|---|
| No evals yet | Start with a CI gate (Promptfoo or DeepEval) on 20 to 50 hand-picked cases, then add production evals once you have real traffic |
| CI evals pass but users still complain | Add production evals; your dataset is missing the real failures |
| You want one vendor for both halves | Braintrust, LangSmith, Langfuse, Arize AX, Opik, or Confident AI |
| You want open source for both halves | Langfuse or Opik, self-hosted |
| Failures only show up across whole conversations | Use a tool that judges whole conversations in production (Agnost AI, LangSmith thread evaluators, Arize AX sessions, Opik threads), and feed what it finds into CI |
| You want production failures turned into tested fixes | Agnost AI Auto Improve, with your CI gate on the resulting PR |

For a full comparison of each tool, see [best AI agent evaluation tools](/blog/best-ai-agent-evaluation-tools/). For production tools only, see [best online evaluation tools for AI agents](/blog/best-online-evaluation-tools-ai-agents/), and for rollout steps, [how to set up online evals in production](/blog/how-to-set-up-online-evals-production/).

## Related reading

- [Best AI agent evaluation tools (2026)](/blog/best-ai-agent-evaluation-tools/)
- [Turn production agent failures into regression tests](/blog/production-failures-regression-tests/)
- [What are online evals? Online vs offline evaluation for AI agents](/blog/what-are-online-evals/)
- [How to set up online evals for AI agents in production](/blog/how-to-set-up-online-evals-production/)
- [Evals catch known unknowns. Production catches unknown unknowns.](/blog/evals-known-unknowns-production-data/)
- [Your agent is failing in production even when your evals pass](/blog/agent-failing-production-evals-pass/)

## Frequently asked questions

### Should I run LLM evals in CI or on production traffic?

Both, because they catch different failures. CI evals run a fixed dataset on every change and block regressions on cases you already know about. Production evals score real user traffic and find failures you didn't know to test for: new intents, drift, and problems that only appear several turns into a conversation. The two connect: every confirmed production failure becomes a new CI regression case.

### What do CI evals catch that production evals don't?

CI evals catch regressions before users see them. When you change a prompt, model, tool, or retrieval setting, a CI eval reruns known cases and fails the build if something that used to pass now fails. Production evals only see a problem after it has reached users.

### What do production evals catch that CI evals miss?

Failures that aren't in your dataset. Real users ask things you didn't anticipate, upstream data and model behaviour drift, tools return errors you never mocked, and multi-turn conversations go wrong in ways a single-turn test case can't reproduce. Production evals judge real traffic, so they surface these unknown failures.

### How do I add an LLM eval gate to GitHub Actions?

With Promptfoo, run npx promptfoo@latest eval -c promptfooconfig.yaml in a workflow triggered on pull_request. promptfoo eval exits with code 100 when a test fails or the pass rate falls below PROMPTFOO_PASS_RATE_THRESHOLD, which fails the job. With DeepEval, write pytest-style test files and run deepeval test run in the workflow. Braintrust (eval-action), Langfuse (experiment-action), LangSmith (pytest, Vitest/Jest), Opik (PyTest) and Arize AX (GitHub Actions guide) also document CI setups.

### Which LLM eval tools do both CI and production evals?

Braintrust, LangSmith, Langfuse, Arize AX, Opik and Confident AI (with DeepEval) document both a CI path and online evaluation on production traffic. Promptfoo, DeepEval on its own and Ragas are CI or offline only. Agnost AI is production only: it judges whole production conversations and does not do CI gating.

### How do I turn production failures into CI regression tests?

When a production eval flags a failure, export the conversation, write down the expected behaviour, and save it as a test case with the inputs and tool responses the agent saw. Add it to the dataset your CI eval runs. Once the fix merges, the case stays in the suite so the failure can't come back unnoticed.

### How many test cases does a CI eval need?

Enough to cover the behaviours you can't afford to break, and small enough to run on every pull request. Most teams start with a few dozen hand-picked cases, then grow the set mainly from confirmed production failures rather than from synthetic examples.
