How do you evaluate multi-turn agent threads where success depends on the whole conversation, not one response?

Most agents don’t fail on a single response—they fail because the overall conversation drifts, loses context, or quietly violates constraints over time. If you only score final answers or isolated turns, you’ll miss the failures that actually matter in production. Evaluating multi-turn agent threads is about testing: did the agent achieve the user’s goal across the entire conversation, given all the branching logic, tools, and state involved?

Quick Answer: You evaluate multi-turn agent threads by treating each conversation as a “thread,” running thread-level evaluators once the full interaction completes, and scoring whether the agent maintained context, respected constraints, and achieved the end goal. LangSmith does this trace-first: it groups all runs into threads, lets you define multi-turn evaluators (LLM, rule-based, or human), and turns failing threads into datasets you can iterate on.

---

The Quick Overview

• What It Is: Multi-turn thread evaluation is a way to test agents on full conversations instead of single responses, so you can measure whether they carry context, manage state, and reach user goals across multiple turns.
• Who It Is For: Teams building agents that do more than answer one-off questions—support bots, research copilots, workflow agents, internal tools where correctness and policy adherence matter over long interactions.
• Core Problem Solved: Single-response metrics hide the real failures. You need to know when an agent “seems fine” per turn but fails the user across the whole thread—e.g., missing steps, ignoring history, or breaking policy on turn 9 of 12.

---

How It Works

At a high level, you:

1. Capture and group traces into threads.
   Every user–agent interaction (plus internal tool calls) is traced. LangSmith groups these traces into threads representing one conversation or session.

2. Define what “success” means at the thread level.
   You define success criteria for the whole conversation: goal completion, constraints, tone, safety, or process adherence across all turns and tools.

3. Run multi-turn evaluators once threads complete.
   When a thread is done, you run evaluators that see the entire thread—multi-turn LLM-as-judge, rule-based checks, or human review via annotation queues. Failing threads become datasets for offline experimentation and regression testing.

Let’s break down the phases.

1. Capture threads, not just runs

In production, your agent is executing many runs: prompts, tool calls, sub-agents, retries. On their own, those runs are noisy; you can’t tell whether the overall conversation worked.

Thread-level evaluation starts with grouping:

• Multi-turn context: All messages between user and agent in chronological order.
• Runs and tools in context: Every internal step, tool call, and sub-agent invocation tied to the turn that triggered it.
• State evolution: How memory, files, or other artifacts changed across turns.
• Time span: Conversations that may last minutes, hours, or days.

LangSmith does this automatically. Every trace is attached to a thread ID, so you can replay the full path: what the user said, how the agent responded, what tools it called, and where it went off the rails.

2. Define thread-level success criteria

You can’t evaluate threads if “success” is fuzzy. For multi-turn threads, success almost always depends on the whole arc:

• Goal achievement: Did the agent actually accomplish the task the user cared about?
• Context retention: Did it remember and correctly use prior turns?
• Constraint adherence: Did it respect instructions like “don’t send emails” or “never show internal IDs” for the entire thread?
• Trajectory quality: Did it take a reasonable path—no dead loops, unnecessary tool calls, or contradictory messages?

Examples:

• Support agent:
  • Success = resolved the ticket, no policy violations, user sentiment improved.
• Research agent:
  • Success = correct final answer, sources cited, all user clarifications integrated.
• Workflow / ops agent:
  • Success = all required steps done, tool actions approved correctly, no forbidden tools used.

You encode this as evaluation prompts, rule-based checks, or annotation guidelines.

3. Use the right granularity: single-step, full-turn, and multi-turn

You shouldn’t throw away your existing evals. Multi-turn evaluation sits on top of the stack:

• Single-step evaluation:
  Unit tests for a specific decision point (e.g., “When user mentions ‘refund’, did the router pick the ‘billing’ tool?”). Great for debugging routing, tool selection, or safety gating.

• Full-turn evaluation:
  Scores a single user–agent exchange. You might check answer correctness, style, or safety per message.

• Multi-turn evaluation (threads):
  Validates the entire conversation: did the agent maintain context, make consistent decisions, and complete the user’s goal? This is where you catch failures that only emerge across a sequence of turns.

LangSmith supports all three patterns. For multi-turn, the key is that evaluators operate on threads, not just messages.

---

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Thread-level tracing	Groups all runs, turns, and tool calls into a single conversation thread.	Lets you see exactly what happened, in what order, and why a multi-turn eval passed or failed.
Multi-turn evaluators	Run LLM, rule-based, or custom evaluators once a thread completes.	Measures whether the agent achieved the user’s goal across the whole conversation, not just per reply.
Trace-to-dataset workflow	Converts failing threads into reusable datasets for offline experimentation.	Turns production failures into test cases so you can prevent regressions before deploying new versions.

---

Ideal Use Cases

• Best for support and CX agents:
  Because they resolve real tickets over several turns, often with follow-up questions and branching flows. Thread-level evaluation lets you test “was this ticket actually resolved within policy?” instead of “was each reply polite?”.

• Best for workflow and operations agents:
  Because they orchestrate tools and approvals over time—e.g., updating records, creating orders, scheduling shipments. Thread-level evaluation checks whether the full workflow completed safely and correctly.

---

Limitations & Considerations

• Eval quality is only as good as your spec:
  A vague “rate this conversation 1–10” prompt isn’t enough. You need clear rubrics and few-shot examples. With LangSmith’s Align Evals, you can calibrate LLM evaluators using real human corrections and examples, but you still need to define the criteria.

• Multi-turn evals are heavier than single-turn:
  Evaluating entire threads costs more (token-wise and compute-wise). In practice, teams:

  • Run online multi-turn evals on a sample of production threads.
  • Use offline evals on curated datasets for regression testing before deployment.
  • Set retention and sampling strategies so they don’t evaluate every trivial chat.

---

Pricing & Plans

You evaluate multi-turn threads using LangSmith’s tracing, datasets, and evaluators. Pricing is built around seats plus usage, so teams can start small and scale as they move more agents into production.

• Team / Business plans: Best for product and platform teams that need:

  • Multiple seats (PM, engineer, DS, QA).
  • Trace-based observability across dev and staging.
  • Online/offline evals, including multi-turn.
  • Reasonable retention defaults without managing infra.

• Enterprise plans: Best for larger orgs and regulated environments needing:

  • US/EU data residency, hybrid or self-hosted deployment.
  • SSO/SAML, SCIM, RBAC/ABAC, audit logs.
  • Higher trace volume, longer retention (up to ~400 days), and strict data governance (LangSmith does not use your data to train models).
  • Support for complex agent fleets and custom evaluators at scale.

You can connect any agent stack (Python, TypeScript, Go, Java, OpenTelemetry) and bring your own models; LangSmith is framework-agnostic.

---

Frequently Asked Questions

How is multi-turn evaluation different from just scoring each response individually?

Short Answer: Multi-turn evaluation scores the entire conversation as a unit—whether the agent achieved the user’s goal across all turns—rather than scoring each response in isolation.

Details:
Per-message scoring is useful but blind to trajectory. An agent can:

• Give four “correct enough” answers but still fail to actually solve the problem.
• Follow instructions on early turns, then quietly ignore them later.
• Loop or oscillate (“Let me re-check…”) without making progress.

Multi-turn evaluation:

• Takes the full thread (all turns, tool calls, and state changes) as input.
• Asks evaluators questions like “Did the agent fully resolve the user’s request?” “Did it maintain context and respect constraints throughout?”
• Reveals pattern failures—like losing context after long sequences, misusing memory, or violating policies only after multiple clarifications.

In LangSmith, you can run both: per-turn evals for debug, multi-turn thread evals for end-to-end quality.

---

How do I set up multi-turn evaluation for my agent in LangSmith?

Short Answer: Instrument your agent with tracing, group runs into threads, define a multi-turn evaluator, and then run it online in production or offline on datasets of conversations.

Details:

The workflow looks like:

1. Instrument tracing:
   Use LangSmith’s SDKs (Python, TypeScript, Go, Java) or OpenTelemetry integration to send runs. Make sure each conversation carries a thread or session ID so LangSmith can group them.

2. Confirm threads in the UI:
   In LangSmith, inspect a few conversations to ensure:

   • All turns between user and agent show up in one thread.
   • Tool calls and sub-runs are nested correctly under each turn.

3. Define a multi-turn evaluator:

   • LLM-based: A prompt that reads the full thread and answers questions like:
     • “Did the agent achieve the user’s goal? (yes/no)”
     • “Did the agent respect this constraint: [policy]?”
   • Rule-based: Scripts that check for required steps, forbidden actions, or state transitions.
   • Human: Annotation queues where reviewers score threads against a rubric.

4. Run offline evals first:

   • Pull a dataset of representative threads (successes, failures, edge cases).
   • Run your multi-turn evaluator and inspect disagreements.
   • Iterate on the prompt/rules until the evaluator matches human judgment.

5. Turn on online evaluation:

   • Configure LangSmith to run your multi-turn evaluator when threads complete in production.
   • Sample traffic (e.g., 5–10% of threads) to control cost.
   • Use dashboards to track scores over time, then drill into failing threads to debug.

From there, you can convert failing threads into datasets and run A/B comparisons between agent versions before you roll them out.

---

Summary

Evaluating multi-turn agent threads is about measuring the thing that actually matters in production: did the agent complete the user’s goal over the entire conversation while staying in bounds? Single-step and per-turn evals are necessary but not sufficient; you need thread-level evaluation to catch context loss, policy drift, and trajectory issues that only appear over time.

LangSmith makes that practical by:

• Tracing all runs and grouping them into threads.
• Letting you define multi-turn evaluators—LLM, rule-based, or human.
• Running them online and offline, then turning failing conversations into datasets for regression tests.
• Providing the observability, governance, and deployment controls to ship agents that work—and keep working—as you iterate.

---

Next Step

Get Started