How do teams turn real production agent conversations into a repeatable test set without manually curating logs?

Most teams discover the real failure modes of their agents in production, not in staging. The hard part is turning those messy, long, branching conversations into a clean, repeatable test set—without asking engineers to trawl through logs all day. The good news: if you treat traces as first-class data, you can automate most of that pipeline.

Quick Answer: You turn production conversations into a repeatable test set by tracing every agent run, sampling the right failures and wins, and pushing them into curated datasets with automation and light human review. LangSmith does this end-to-end: capture traces → mine them for patterns → convert into datasets → evaluate and re-run on every change.

---

The Quick Overview

• What It Is: A trace-first workflow that converts real production agent conversations into structured, reusable test sets with minimal manual log review.
• Who It Is For: Teams running LLM agents in production—support, ops, research, product, and platform teams—who need to improve quality without burning time on log spelunking.
• Core Problem Solved: Production is the only place you see the full range of user behavior, but raw logs are noisy. You need a repeatable way to capture the right examples, label them, and reuse them as tests whenever you change prompts, tools, or models.

---

How It Works

The foundation is simple: every agent run becomes a trace with complete context. From there, you selectively turn those traces into a test set.

At a high level, the workflow looks like this:

1. Instrument and capture: Log every agent run as a structured trace with inputs, tool calls, intermediate states, and outputs.
2. Filter and sample: Use queries, analytics, and agents to surface high-signal conversations—failures, edge cases, and representative wins.
3. Label and lock: Turn those traces into datasets with ground-truth labels or quality scores, so you can re-run them on demand during development and before deploys.

In LangSmith, you move through this loop continuously: production traces fuel new tests, tests feed evals, and evals gate what you ship.

---

Step 1: Capture production conversations as structured traces

Treat traces—not logs—as the source of truth

For traditional apps, code is usually the best record of behavior. For agents, it’s the trace. Every run can take a different path through tools, prompts, and branches, even if the code doesn’t change.

So the first step is to instrument your agent stack so that every conversation is:

• Captured as a run: start time, end time, status, latency.
• Broken into steps: model calls, tools, retrievers, sub-agents.
• Threaded into conversations: each user session forms a coherent history.
• Enriched with metadata: user segment, environment (prod/staging), flags, AB variant, etc.

With LangSmith, you:

• Use SDKs in Python, TypeScript, Go, or Java to instrument your app, or
• Plug in via OpenTelemetry or built-in integrations for popular agent frameworks.
• Get immediate message threading so every multi-turn conversation is grouped and replayable.

This gives you a searchable corpus of real conversations and execution timelines, not just free-form logs.

---

Step 2: Filter and sample the right traces automatically

Once everything’s traced, the next challenge is volume. You might have millions of conversations per week. You don’t want humans scrolling through all of them.

Use analytics to focus on high-signal runs

You start by narrowing down to the conversations that matter for testing:

You can filter by:

• Outcome signals
  • Explicit thumbs up/down or satisfaction scores
  • “Escalated to human” events
  • Refunds, reopened tickets, negative CSAT
• Operational signals
  • Long latency, timeouts, tool failures
  • High token usage
  • Many tool calls or loops in a single conversation
• Segment and feature flags
  • Only production runs (not test or sandbox)
  • Specific products, intents, or experiments (e.g., “beta_booking_agent_v2”)

In LangSmith, you can:

• Slice by status, latency, error types, tool usage, and metadata.
• Use dashboards and analytics to find patterns—tools that often precede failures, prompts that correlate with escalations, etc.
• Set up alerts when certain error rates or negative signals spike, and automatically route those traces into review queues.

Let agents pre-select and summarize

You can also use AI itself to help you mine this data:

• Have an internal evaluator agent scan traces to:
  • Flag confusing conversations or hallucinations.
  • Summarize what went wrong and why.
  • Suggest whether a trace should be added to a test set.
• Use this to propose candidate test cases for a human to approve instead of starting from raw logs.

LangSmith’s Insights Agent operates exactly in this direction: it sits on top of your traces and surfaces patterns, anomalies, and candidate examples for closer inspection.

---

Step 3: Turn selected traces into reusable datasets

Once you have a subset of high-signal conversations, you convert them into a stable test set.

Convert runs into datasets with one click or API

From a filtered view of traces, you can:

• Bulk-select runs that match your criteria.
• Export them as a dataset directly in LangSmith or via API.
• Store:
  • Inputs (user message or full conversation history).
  • Context (tools used, environment, metadata).
  • Model output or agent action sequence.

These datasets become your “frozen” view of real conversations at a point in time, ready for evaluation.

Add ground truth or quality labels

To make the test set meaningful, you attach labels. That can be:

• Exact ground truth (for tasks with a single correct answer):

  • SQL queries, API payloads, structured responses.
  • Normalized answers for Q&A with known truths.

• Graded quality labels (for subjective or open-ended tasks):

  • 1–5 helpfulness scores.
  • Binary pass/fail against your policy.
  • Error types (missing context, policy violation, hallucination).

You don’t want to do this entirely by hand. Instead:

• Use LLM-as-judge evaluators to auto-score outputs.
• Calibrate them with human-labeled examples so they match your bar.
• Route tricky or high-risk examples to annotation queues for SME review.

LangSmith’s Align Evals is built specifically for this calibration loop: start with human corrections and few-shot examples, tune your evaluators, and then let them handle the bulk of labeling at scale.

---

Step 4: Cover single-step, full-turn, and multi-turn behavior

Production runs let you test different levels of behavior that static test prompts can’t cover.

With your dataset, you can define evals for:

• Single-step decisions

  • Was this tool call correct given the context?
  • Did the agent choose the right retrieval query?
  • Did it interpret the user’s intent properly?

• Full-turn performance

  • Given this input and context, was the final answer correct, helpful, and safe?
  • Did it follow output formats and internal policy?

• Multi-turn behavior

  • Did the agent maintain context across a long conversation?
  • Did it remember constraints and preferences from earlier turns?
  • Did it avoid re-asking questions or contradicting itself?

LangSmith supports multi-turn evals so you can replay entire conversations, not just one-off prompts. That’s where most agents fail in production—and it’s exactly what synthetic “single-shot” test sets often miss.

---

Step 5: Run offline evals on every change

Once you’ve turned live traces into structured datasets, you can use them as part of your development lifecycle.

Treat datasets like unit tests for agents

Whenever you:

• Change a prompt or chain logic,
• Swap models,
• Add or remove tools,
• Adjust routing strategies,

you can:

• Run your candidate versions against the same production-derived test set.
• Compute scores across dimensions:
  • Task success, factuality, compliance.
  • Latency and cost.
• Compare side-by-side to see regressions before you ship.

LangSmith:

• Lets you run offline evals across datasets at will.
• Visualizes runs and metrics per model version, prompt variant, and configuration.
• Makes it easy to roll back if a change degrades performance on your critical examples.

This is how you make production your “primary teacher” without turning your users into unpaid QA.

---

Step 6: Close the loop with production feedback

The real power comes from treating this as a continuous loop, not a one-off export.

You can:

• Auto-create tests from incidents: When an engineer or SME spots a bad conversation in LangSmith, they can click “Add to dataset” so that failure is never allowed to sneak back in.
• Sample regularly: For high-volume agents, periodically sample fresh traces (e.g., weekly) into a “drift watch” dataset to check for performance changes as user behavior shifts.
• Tune evaluators over time: As humans correct evaluator mistakes, feed those back into Align Evals so auto-judging gets sharper.

Over time, your test suite increasingly reflects your actual production surface area, not just what you guessed up front.

---

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Trace capture and message threading	Logs every agent run with full context and groups turns into conversations.	Gives you a complete, replayable record of real interactions.
Dataset creation from traces	Converts selected runs into structured datasets with inputs and outputs.	Lets you reuse real conversations as stable test sets.
Calibrated evaluators and Align Evals	Uses LLM-as-judge plus human calibration to label and score outputs.	Scales evaluation without losing domain-specific accuracy.
Multi-turn and offline evals	Replays full conversations and scores behavior across turns.	Catches regressions in context retention and complex flows.

---

Ideal Use Cases

• Best for quality-focused agent teams: Because it lets you convert real failures and edge cases into permanent tests, you prevent the “whack-a-mole” bug cycle every time you tweak prompts or models.
• Best for regulated or high-stakes domains: Because you can encode compliance rules and SME-reviewed labels into evals, then run them on every production-derived dataset before deploying changes.

---

Limitations & Considerations

• You still need some human review: Automated evals go far, but high-risk or nuanced domains (finance, medical, legal) require SMEs to calibrate and spot-check. The goal is to use tools so humans label the 5–10% that matter most, not everything.
• Garbage-in, garbage-out risk: If you only sample from “happy path” conversations or ignore negative signals, your test set will miss real failure modes. Be deliberate about including difficult, long, and escalated threads.

---

Pricing & Plans

LangSmith is designed for teams that care about trace volume, retention, and collaboration, with seat-based access plus pay-as-you-go usage.

Typical structure:

• Team / Growth plan: Best for product and platform teams needing collaborative tracing, datasets, and evals across a few agents and environments.

  • Multiple seats for engineers, data scientists, and PMs.
  • Generous included traces and standard retention (e.g., 14+ days).
  • Great for getting a first production agent into a measured, testable state.

• Enterprise plan: Best for large organizations needing long-term retention, strict governance, and integration into existing infra.

  • US/EU data residency, hybrid, or self-hosted/VPC options.
  • SSO/SAML, SCIM, RBAC/ABAC, audit logs, and encryption controls.
  • Extended trace and dataset retention (e.g., 400-day), high-volume pricing, and dedicated support.

To get precise pricing and options, talk to the LangChain team directly.

---

Frequently Asked Questions

Do I need to export and clean logs manually to build these test sets?

Short Answer: No. If you trace your agents correctly, you can create datasets directly from structured runs—no manual log scraping required.

Details: With LangSmith, agent runs are already structured as traces with inputs, outputs, and metadata. You filter runs (e.g., all escalated conversations for a specific tool), bulk-select them, and click “Create dataset.” You can automate this further via API or workflows that push certain runs (like negative feedback or policy violations) into datasets or annotation queues. The only manual work that remains is targeted labeling and calibration, not raw log processing.

---

How do we keep the test set fresh as user behavior changes?

Short Answer: Continuously sample new production traces into datasets and re-run evals so your tests track real-world usage.

Details: User language and edge cases change over time, especially for public-facing agents. In LangSmith, you can periodically:

• Sample traces from recent weeks based on usage, failure, or drift signals.
• Add them to existing datasets or create “drift watch” datasets.
• Auto-evaluate with your calibrated evaluators and review outliers. When a new failure pattern appears, you add a few representative traces to your critical test suites. That way, every regression you see in production becomes a permanent guardrail for future changes.

---

Summary

The way teams turn real production conversations into a repeatable test set—without manually curating logs—is by going trace-first. Instrument every agent run, use analytics and agents to surface high-signal examples, convert those traces into datasets, and layer calibrated evals on top. LangSmith wraps this into a tight loop: trace capture, dataset creation, offline and multi-turn evals, and deployment with rollbacks.

Over time, your test suite stops being hypothetical prompts and becomes a direct mirror of what actually happens in production. That’s how you ship agents that work—and keep them working—as models, prompts, and user behavior evolve.

---

Next Step

Get Started