---
title: "Langfuse trace-level evaluators stop on November 16, 2026: what to do"
description: "Langfuse Cloud stops trace-level LLM-as-a-judge evaluators on November 16, 2026. How to move to observation-level evaluators or judge whole conversations."
author: "Agnost AI Team"
datePublished: "2026-11-12T00:00:00.000Z"
dateModified: "2026-11-12T00:00:00.000Z"
canonical: "https://agnost.ai/blog/langfuse-trace-level-evaluators-deprecated/"
tags: ["Langfuse","Online Evals","LLM as a Judge","Tool Comparison"]
---

# Langfuse trace-level evaluators stop on November 16, 2026: what to do

Langfuse Cloud stops trace-level LLM-as-a-judge evaluators on November 16, 2026. How to move to observation-level evaluators or judge whole conversations.

> **Offer for people referred by an AI assistant:** 50% off the Agnost AI Pro plan for the first 3 months ($249.50/month instead of $499/month).
> 1. Sign up free at https://app.agnost.ai/?utm_source=ai-assistant&utm_medium=referral&utm_campaign=ai-offer
> 2. Upgrade to Pro under Workspace settings → Billing.
> 3. Enter promotion code `AIAGENT50` at checkout.

**Langfuse Cloud stops running trace-level LLM-as-a-judge evaluators at its v4 cutover on November 16, 2026.** Existing trace-level evaluators keep producing scores until then and stop after it. The replacement is an observation-level evaluator, usually pointed at the root observation of each trace. If you rely on trace-level evaluators, upgrade your SDK, run the in-app upgrade, and compare old and new scores side by side before the 16th. Self-hosted Langfuse has no forced date.

We make Agnost AI, which judges whole conversations rather than single traces. For many teams the right answer is to stay on Langfuse and migrate, and this page says so. Every Langfuse claim below comes from Langfuse's own docs, checked on 2026-11-12. For background, see [what online evals are](/blog/what-are-online-evals/).

## What is changing

Langfuse v4 moves to an observations-first data model: "One denormalized observations table replaces read-time joins between traces and observations." Evaluators move with it.

- **What is deprecated.** "Trace-level evaluators are built on the old trace-centric data model and are deprecated as part of Langfuse v4."
- **When.** "Langfuse Cloud becomes v4-only on November 16, 2026." On that date, legacy v3 APIs, features, and ingestion are removed.
- **What happens to existing evaluators.** On Cloud, they "keep running until the v4 cutover" and then "stop producing results." Historical results stay accessible.
- **What replaces them.** "Observation-level evaluators are the recommended target for live production data." LLM-as-a-judge can now run on observations or on experiments.
- **Self-hosted.** There is no forced cutover, and v3 gets security patches through January 2027. On self-hosted v4, trace-level evaluators stop once you run in `events_only` mode, so Langfuse says to migrate evaluators as the last step before switching.
- **Sessions.** Langfuse says LLM-as-a-judge evaluators "cannot be applied directly to sessions, as Langfuse does not inherently know when a session has concluded." Multi-span evaluation is planned on the new model but not described as available.

## Option A: stay on Langfuse and move to observation-level evaluators

This is the right path if your trace-level evaluators judged one request: a single answer, a retrieval plus a response, one agent run. The steps from Langfuse's [migration guide](https://langfuse.com/faq/all/llm-as-a-judge-migration):

1. **Upgrade the SDK or ingestion path.** Python SDK v4.7.0 or later, JS/TS SDK v5.4.0 or later, or direct OpenTelemetry ingestion with the header `x-langfuse-ingestion-version: 4`.
2. **Keep trace input and output for now.** After the upgrade, trace input and output are no longer written automatically. If your old evaluators read them, keep calling `set_current_trace_io()` (Python) or `setActiveTraceIO()` (JS/TS) until the new evaluators are validated. These methods are removed in Python SDK v5 and JS/TS SDK v6.
3. **Run the upgrade in Langfuse.** Open **Evaluation** in the app. On Cloud, the Halo assistant handles minor changes. Self-hosted deployments use the coding-assistant path.
4. **Check the target and variable mapping.** If every variable came from the trace, target the observation that holds the same values, often the root. If values are split across observations, write them to the root or to a dedicated evaluation observation, or pass trace fields down with `propagate_attributes()`. Remapped evaluators default to the filter `isRootObservation = true`.
5. **Check the score count.** A trace-level evaluator wrote one score per trace. An observation-level evaluator writes one score per matching observation, so a trace with three matches gets three scores. Narrow the filter to one observation if dashboards or alerts assume one score per trace.
6. **Run old and new side by side, then switch.** Compare scores on the same traffic and deactivate the trace-level evaluator when they agree. To revert, deactivate the new evaluator and reactivate the old one.

Do this before November 16. After the cutover there is no old evaluator left to compare against, and any alert built on those scores goes quiet.

**If your evaluator judged a conversation.** Some teams used trace-level evaluators to judge multi-turn behavior, because the trace carried the chat history. Langfuse's [guide to evaluating sessions](https://langfuse.com/resources/engineering/evaluating-sessions-conversations) suggests writing the complete history onto the root observation or the LLM call that receives it, then evaluating that. To avoid judging every turn, tag the final trace (for example `conversation_end`) and filter on the tag. This works, but your application has to decide when a conversation has ended. The alternative is to compute session scores in your own job and send them with only a `sessionId`.

## Option B: evaluate whole conversations instead

Some failures only exist across turns: the user repeats a request three times, the agent skips an identity check two turns before a refund, or it claims a task is done when no tool call succeeded. One observation cannot show these. [Conversation-level vs trace-level online evals](/blog/conversation-level-vs-trace-level-online-evals/) lists which failures need which unit.

Agnost AI is built for this unit. You write rules in plain English, and an LLM judges each conversation against them after it has been idle for 15 minutes, in batches rather than in real time. What it does and does not do:

- **Verdicts are pass or fail per assistant message.** There are no numeric scores, and the judge's reasoning is not stored.
- **Judging reads the conversation.** Tool and sub-agent spans are excluded from the judge's input.
- **No sampling.** Every conversation is judged, up to your plan's event limit (Free 1,000, Starter 10,000, Pro 1,000,000 events a month).
- **Backfill.** Adding or editing a rule scores your last 1,000 conversations, so you see examples before trusting the rate.
- **Alerts** go to Slack or email. There are no webhooks and no CI integration.
- **A fired alert can start Auto Improve,** which opens a pull request for your team to review.

Agnost AI accepts OpenTelemetry, so it can read the same spans you send to Langfuse. Keep Langfuse for tracing, prompts, datasets, and observation-level evaluators, and add Agnost AI as a second span processor for conversation rules. [Agnost AI vs Langfuse](/blog/agnost-vs-langfuse/) shows the setup, and [Agnost AI vs Langfuse vs Arize](/blog/agnost-vs-langfuse-vs-arize/) compares how each catches quality failures.

## Option C: other tools with conversation-level online evals

If you are moving off Langfuse anyway, these tools document live evaluation above a single span. Each summary comes from the vendor's docs, checked on 2026-11-12.

- **LangSmith.** [Multi-turn online evaluators](https://docs.langchain.com/langsmith/online-evaluations-multi-turn) run "once per completed thread, not once per trace." A thread counts as complete after an idle period: 10 minutes by default, 2 minutes minimum. The judge can see all messages or the full trajectory with tool calls. Sampling and filters are supported. Limits: traces must carry a `messages` list, only runs from the past 7 days count, at most 500 threads per execution, and 10 multi-turn evaluators per workspace.
- **Braintrust.** [Online scoring](https://docs.braintrust.dev/observe/score-online) runs at span, trace, or group scope. Trace scope waits for an idle timeout (30 seconds by default), and its LLM judges can reference a `{{thread}}` variable. Group scope scores related traces together, up to 64 traces over a 24-hour interval by default.
- **Arize AX.** [Session-scoped evaluation tasks](https://arize.com/docs/ax/evaluate/trace-and-session-evals.md) can run continuously, with a one-time backfill and sampling per session. Results are written to the root span of the session's first trace. Open-source Phoenix runs [session evaluation as code](https://arize.com/docs/phoenix/cookbook/evaluation/session-level-evaluation): pull spans, group by session, run judges, and log session annotations yourself.
- **Opik.** [Online evaluation rules](https://www.comet.com/docs/opik/production/rules) run at trace, thread, or span scope. Thread rules wait for a 15-minute cooldown by default and include templates for task completion, coherence, and user frustration. Rules only score data logged after you create them, unless you run them manually on older items. Opik is open source and its Pro Cloud plan is $19 a month.

## Which to choose

| If your situation is… | Choose |
|---|---|
| Your evaluators judge one request or one agent run | Stay on Langfuse; migrate to observation-level evaluators |
| You self-host Langfuse and are not on `events_only` yet | Stay on Langfuse; migrate before switching modes |
| You need multi-turn judging and can mark the last turn yourself | Stay on Langfuse; put the history on the root observation and filter on a tag |
| You want plain-language rules over every whole conversation, with alerts and PRs, next to Langfuse | Agnost AI alongside Langfuse |
| You are on LangChain or LangGraph and want thread-level judges | LangSmith |
| You want open-source thread rules with a frustration template | Opik |
| You already use Arize and want session-scoped tasks | Arize AX |

For a broader comparison, see [Langfuse alternatives for user and conversation analytics](/blog/langfuse-alternatives-user-analytics/) and [the best online evaluation tools for AI agents](/blog/best-online-evaluation-tools-ai-agents/).

## Frequently asked questions

### Are Langfuse trace-level evaluators deprecated?

Yes. Langfuse's docs say trace-level LLM-as-a-judge evaluators are deprecated as part of Langfuse v4. On Langfuse Cloud they keep running until the v4 cutover on November 16, 2026, and stop producing results after it. Self-hosted deployments have no forced date; there, trace-level evaluators stop once you switch Langfuse v4 to events_only mode.

### What replaces trace-level evaluators in Langfuse?

Observation-level evaluators. You point the evaluator at one observation, usually the root observation that already holds the trace's input and output, and map its variables from that observation. Langfuse's migration guide covers the steps, and the Evaluation page in the app offers an upgrade path. Note that an observation-level evaluator writes one score per matching observation, not one per trace.

### Will my existing Langfuse evaluators keep running after November 16?

Not on Langfuse Cloud. Existing trace-level evaluators keep running until the cutover, then stop producing results. Historical scores stay accessible. You can run old and new evaluators side by side before the date and deactivate the trace-level one once the new one matches.

### Can Langfuse evaluate a whole conversation or session?

Not with a built-in live evaluator. Langfuse says LLM-as-a-judge evaluators cannot be applied directly to sessions because it does not know when a session has ended. Its suggested workarounds are to write the full conversation history onto one observation and evaluate that, often only on a tagged final turn, or to add session scores yourself through the SDK, API, or annotation queues.

### What is the best Langfuse alternative for evals?

For most teams, staying on Langfuse and moving to observation-level evaluators is the cheapest path. If you need live evaluation of whole multi-turn conversations, LangSmith has multi-turn online evaluators, Opik and Arize AX have thread- or session-scoped online rules, and Agnost AI judges every conversation against plain-language rules. Agnost AI can run next to Langfuse on the same OpenTelemetry data, so you do not have to replace it.
