How do we run multi-turn/thread-level evaluations in LangChain LangSmith (not just single responses)?

Most teams start evaluating agents by scoring single responses. That works for simple Q&A, but it breaks as soon as you introduce memory, tools, or branching logic. To understand whether your agent actually achieved the user’s goal, you need to evaluate the entire thread — the full multi-turn interaction — not just the last message.

This guide walks through how to run multi-turn, thread-level evaluations in LangChain LangSmith, how they differ from single-response evals, and how to wire them into both experimentation and production.

Quick note on terminology:

• Run = a single execution unit (LLM call, tool call, chain, or agent step)
• Thread = a full conversation or workflow spanning many runs
• Multi-turn / thread-level eval = an evaluator that looks at the entire thread trajectory

The Quick Overview

• What It Is: Multi-turn/thread-level evaluation in LangSmith lets you assess the complete conversation or workflow an agent runs through, not just individual model outputs.
• Who It Is For: Teams building agents with memory, tools, or multi-step reasoning that need to verify overall outcomes, adherence to constraints, and interaction quality.
• Core Problem Solved: Single-response scoring misses failures that only show up across steps — context loss, tool misuse, loops, or subtle policy violations. Thread-level evals surface those.

How Thread-Level Evaluation Works in LangSmith

LangSmith treats evaluation as its own first-class workflow: you define evaluators, attach them to datasets or production traffic, and let them score entire traces once a thread completes.

At a high level:

1. Capture full traces for each thread.
   Instrument your agent (Python, TypeScript, Go, Java, OpenTelemetry, or LangChain/LangGraph native) so every step — LLM call, tool, branch — is recorded as a trace.

2. Define multi-turn evaluators.
   Implement evaluators that:

   • Receive the complete thread (all messages and runs)
   • Optionally receive reference data (ground truth, policies, expectations)
   • Return scores and feedback at the thread level

3. Run evaluators offline or online.

   • Offline: Run evals on datasets and historical traces to compare agent versions before deployment.
   • Online: Attach evaluators to production threads so they score once a thread completes (multi-turn online evaluators).

The outcome: for each conversation, you get structured metrics like “Goal Achieved,” “Constraint Compliance,” “Multi-step Reasoning Quality,” plus freeform feedback, all linked to the underlying trace so you can click in and debug where things went wrong.

Step-by-Step: Setting Up Multi-Turn Evaluations

1. Capture rich traces for each conversation

Multi-turn evals only work if you can see the full trajectory.

In your agent code, instrument LangSmith tracing so each user conversation becomes a thread with nested runs:

from langsmith.run_helpers import traceable

@traceable  # or manual run creation if you want fine-grained control
def agent_step(state, user_message):
    # your agent logic here
    ...

Use:

• LangChain/LangGraph: built-in LangSmith integration.
• Other frameworks / custom stack: LangSmith SDKs (Python, TS, Go, Java) or OpenTelemetry to send traces.
• Make sure:
  • Each interaction (thread) uses a consistent thread_id / root run.
  • Tool calls and sub-agents are nested properly so the trace shows the full tree.

The more accurate the trace, the more reliable your thread-level evals will be.

2. Decide what “thread success” means

Before you write evaluators, clarify what success looks like across a full conversation. Common thread-level objectives:

• Did the agent achieve the user’s goal by the end of the thread?
• Did it maintain context and memory across turns?
• Did it avoid unnecessary loops or redundant tool calls?
• Did it respect policies (PII, tone, compliance) across all messages?
• Did it handle corrections and context switches gracefully?

You’ll turn these into evaluator prompts, heuristics, or human review criteria.

3. Implement multi-turn evaluators

LangSmith supports several evaluator types at the thread level:

• LLM-as-judge multi-turn evaluators
• Heuristic / rule-based checks
• Human evaluation via annotation queues
• Pairwise comparison of two full threads

All of these can operate on the entire conversation once a thread completes.

A. LLM-as-judge multi-turn evaluators

For most teams, this is the starting point: use one model to grade another, but on the full thread.

You define an evaluator that receives:

• The full conversation transcript (user + agent messages, tool outputs if you want)
• Optional reference data (expected answers, policies, metadata)
• A prompt that asks the judge model to score the thread on specific axes

The evaluator returns:

• Numeric scores (e.g., 1–5 or 0–1)
• Optional categorical labels (e.g., PASS/FAIL)
• Freeform feedback / critique

Because LangSmith supports multi-turn online evaluators, you can configure these to run automatically after a thread finishes, not on every individual run.

B. Heuristic and rule-based thread evaluators

Some checks are deterministic and better expressed as code than prompts:

• “Did the agent call this tool more than N times?”
• “Did the agent ever output raw stack traces?”
• “Did it violate a regex-based policy (e.g., exposed SSNs)?”

You can build custom evaluators that:

• Walk the entire trace graph
• Inspect tool calls, arguments, outputs
• Emit scores, booleans, or labels per thread

These play well with LLM-as-judge evaluators — you can combine both in one evaluation run.

C. Human evaluation (annotation queues)

For high-stakes agents, you’ll want humans to review a sample of threads:

• Route full threads into annotation queues
• Give subject matter experts the full timeline and chat transcript
• Have them score:
  • Outcome correctness
  • Policy compliance
  • Experience quality

LangSmith then uses these human labels to:

• Directly measure quality on production traffic
• Calibrate LLM-as-judge evaluators (via Align-style “eval calibration” with human feedback and few-shot examples)

4. Attach evaluators to datasets (offline)

For reproducible, versioned experimentation, use datasets that represent full threads.

1. Build a dataset where each row describes a conversation scenario:

   • Initial user message
   • Optional subsequent turns (scripted or generated)
   • Ground truth outcome or expectations

2. Configure your agent under test to run against this dataset.

3. Attach your multi-turn evaluators so that after each thread completes, LangSmith:

   • Collects the entire conversation trace
   • Passes it to your evaluator(s)
   • Stores scores and feedback alongside the run

You now have offline, thread-level scores you can compare across:

• Different agent strategies
• Prompt or tool changes
• Model swaps

5. Enable multi-turn online evaluators (production)

LangSmith can also run multi-turn evaluators directly on production traffic.

Configure online evaluators that:

• Subscribe to thread completion events
• Pull the entire thread trace
• Run your LLM-as-judge and heuristic checks
• Write results into your evaluation dashboards

With online multi-turn evaluation, you can:

• Detect regressions quickly after a new deployment
• Track experience-level metrics (e.g., “Goal Achieved %”) by segment
• Spot systemic issues like:
  • Context loss after more than N turns
  • Particular tools causing loops
  • Policies being violated in rare edge cases

Because the evaluators are trace-first, you can click from a failing score directly into the exact run timeline to see what happened, in what order, and why.

6. Close the loop with calibration and iteration

The point of thread-level evaluation is not just to score, but to improve.

With LangSmith you can:

• Turn production threads into datasets
  Sample interesting or failing traces and convert them into curated datasets for regression testing.

• Calibrate LLM-as-judge evals
  Use human annotations and few-shot examples to refine judge prompts and thresholds so scores correlate with human judgment.

• Compare versions side-by-side
  Run your multi-turn dataset across:

  • Old vs. new prompts
  • Old vs. new tools
  • Different memory strategies and compare thread-level metrics directly.

• Guard deployments with eval gates
  In CI/CD or deployment workflows, require that:

  • Multi-turn quality metrics meet thresholds
  • No new classes of policy violations appear before shipping to production.

Example Thread-Level Evaluation Patterns

To make this concrete, here are a few patterns teams use.

Pattern 1: “Did the agent solve the user’s problem?”

Use an LLM-as-judge evaluator over the full thread:

• Input:
  • Conversation transcript
  • The user’s original goal (and, optionally, ground truth)
• Prompt criteria:
  • Did the agent provide a final answer that resolves the user’s intent?
  • Did it address necessary follow-up questions?
  • Did it avoid hallucinating unsupported claims?
• Output:
  • goal_achieved_score (0–1)
  • reason (text explanation)

Pattern 2: “Did the agent manage context and memory across turns?”

Evaluate memory handling:

• Inspect whether:
  • Important details from earlier turns are used in later responses
  • The agent contradicts itself or forgets constraints
• Use:
  • LLM-as-judge to reason about consistency
  • Heuristics to check specific flags or session variables in the trace

Pattern 3: “Did the agent behave well as a multi-step workflow?”

For tool-heavy agents:

• Evaluate tool usage:
  • Number of tool calls
  • Loops or repeated failing tools
  • Time to completion
• Use:
  • Heuristic evaluator walking the trace tree
  • LLM-as-judge that reads the step-by-step timeline and scores “efficiency” and “adherence to plan”

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Multi-turn online evaluators	Run evaluators once a thread completes, on the full interaction	Catch failures that only appear across multiple turns or tools, not on single responses
Thread-level trace inspection	Shows the complete run timeline (tools, messages, decisions) for each eval	Lets you trace back from a bad score to the exact step where the agent went off the rails
Human + LLM-as-judge evaluation	Combines annotation queues with automated scoring and calibration	Aligns automated scores with domain experts and keeps evals grounded in real user expectations

Ideal Use Cases

• Best for tool-using, long-context agents:
  Because they often “look fine” at the last response, but actually loop, forget context, or misuse tools in the middle of the thread.

• Best for production agents with compliance constraints:
  Because policy violations and subtle tone issues often surface only after several turns — thread-level evals can detect those patterns reliably.

Limitations & Considerations

• Evaluator quality depends on prompts and data:
  LLM-as-judge evals are only as good as their instructions and calibration. Use human annotations and few-shot examples to tune them and periodically revalidate.

• Cost and latency for online multi-turn evals:
  Scoring every production thread can be expensive. Common patterns:

  • Sample a subset of threads (e.g., 5–10%)
  • Focus on risky segments (new features, specific customer tiers)
  • Use lighter heuristics for broad coverage, LLM-as-judge for deeper slices

Pricing & Plans

LangSmith is priced for teams of any size and follows a pay-for-what-you-use model:

• You get access to both tracing and evaluation on all plans.
• You pay based on usage (e.g., number of traces/events, eval runs) rather than being locked into a specific framework.

Typical patterns:

• Team / Growth plans: Best for product teams moving from prototype to production who need robust multi-turn evals, online/offline scoring, and standard retention.
• Enterprise plans: Best for larger organizations needing:
  • Extended retention (e.g., 400-day traces and eval history)
  • Advanced admin (SSO/SAML, SCIM, RBAC/ABAC, audit logs)
  • Data residency (US/EU), hybrid or self-hosted options
  • The ability to keep all traces and eval data within their own VPC

For exact pricing and feature breakdown, talk to sales — the structure is designed so you can start small and scale eval coverage as your agent footprint grows.

Frequently Asked Questions

How is thread-level evaluation different from just scoring the final answer?

Short Answer: Thread-level evals look at the entire conversation and agent trajectory, not just the last output.

Details:
Single-response evals answer “Was this answer good?” but ignore how the agent got there. Thread-level evaluation considers:

• All turns, not just the final one
• Tool calls, branches, and intermediate steps
• Whether the agent maintained context, respected constraints, and converged efficiently

In LangSmith, you do this by running multi-turn evaluators when a thread finishes. They receive the full trace (timeline + messages), so they can detect issues like context loss, unnecessary loops, or policy drift that simple final-answer scoring will miss.

Can I run multi-turn evaluations in CI/CD before shipping changes?

Short Answer: Yes. You can run multi-turn evals on datasets in CI/CD to gate deployments.

Details:
LangSmith’s evaluation framework is designed to fit into your deployment pipeline:

• Build/curate a dataset of representative threads (including tricky edge cases).
• Attach your multi-turn evaluators (LLM-as-judge, heuristics, or both).
• In CI/CD, run your candidate agent version against that dataset.
• Fail the build or block deployment if thread-level metrics regress (e.g., lower goal achievement, higher policy violation rate).

Because LangSmith is framework-agnostic and accessible via SDKs and APIs, you can integrate these checks into whatever CI system you use (GitHub Actions, GitLab CI, etc.).

Summary

If you’re only scoring single responses, you’re missing most of the failure modes that matter for real agents — context loss, bad tool loops, subtle policy drift, and multi-step reasoning gaps. LangSmith’s multi-turn/thread-level evaluations give you a trace-first way to measure the actual behavior of your agents across full conversations.

By capturing rich traces, defining thread-level evaluators (LLM-as-judge, heuristics, human review), and running them offline on datasets and online in production, you can:

• See exactly what your agent did over time
• Quantify whether it actually solved user goals
• Detect regressions before they impact users
• Continuously improve agent quality with a tight trace → dataset → eval → iteration loop

Next Step

Get Started