How do we create a dataset in LangChain LangSmith from production traces and use it for regression tests?

Most teams don’t start with a perfect test set. You start with real users hitting your agent, real failures in production, and a vague sense that “something regressed.” LangSmith turns that production reality into structured datasets you can reuse for regression tests every time you ship a change.

Quick Answer: In LangSmith, you create datasets directly from production traces by sampling interesting or failing runs, adding them to a dataset with one click, then attaching evaluators and running those datasets as regression suites whenever you change prompts, models, or logic.

The Quick Overview

• What It Is: A workflow in LangSmith to turn production traces into reusable datasets and run them as regression tests for your LLM app or agent.
• Who It Is For: Teams shipping agents into production who need to catch regressions across prompts, models, tools, and workflows—not just unit tests on static inputs.
• Core Problem Solved: You can’t pre-plan every input or branch. LangSmith lets you capture real production behavior, turn it into a dataset, and continuously test against it so quality doesn’t silently drift.

How It Works

LangSmith is trace-first: it records what your agent actually did in production—inputs, outputs, tool calls, intermediate steps. From there, you:

1. Capture production traces.
2. Sample the important or problematic ones into a dataset.
3. Attach evaluators and run that dataset as a regression suite whenever you make a change.

Over time, you build a flywheel: production traffic → traces → datasets → evaluations → fixes → redeploy → more traces.

1. Capture high-fidelity production traces

At minimum you need LangSmith tracing wired into your production stack. LangSmith is framework agnostic, so you can send traces from:

• LangChain or LangGraph
• OpenAI / Anthropic SDKs
• Custom Python / TypeScript / Go / Java agents
• Any other framework using the LangSmith SDK or OpenTelemetry

Mechanically, you:

• Configure the LangSmith SDK with your API key.
• Wrap your LLM calls, tools, or agent orchestration in tracked “runs”.
• Deploy as normal. The SDK ships traces asynchronously to a distributed collector, so your app performance isn’t impacted and your agent keeps running even if LangSmith has an incident.

Each production request becomes a structured trace showing:

• User input and context
• Model prompts and responses
• Tool calls and parameters
• Retrieved documents
• Intermediate agent decisions
• Final output

This is the raw material for your datasets.

2. Discover what to test from production behavior

If you don’t already know which flows to capture, LangSmith gives you multiple ways to surface them:

• Manual exploration: Browse the “Runs” view, filter by tag, model, latency, or status, and manually inspect threads that look suspicious or high-value.
• Online evals (if configured): Flag runs that failed evaluator checks (policy violations, hallucinations, formatting issues) and consider them for inclusion.
• Insights Agent: Use LangSmith’s Insights Agent to summarize usage patterns and identify common workflows, frequent failure modes, or specific cohorts (e.g., a certain tool pattern) worth testing.
• User feedback: Leverage thumbs up/down or explicit feedback flows to find traces where users reported bad answers or surprising behavior.

The goal is not to grab everything. You want representative and high-signal runs:

• Frequent workflows (core product paths)
• Edge cases and known failure modes
• High-value customers or domains
• Long, multi-turn threads where regressions are hard to spot manually

3. Create a dataset from a single trace

When you find a trace that you want to reuse as a test:

1. Open the trace in LangSmith.
2. Review the full run timeline—inputs, intermediate steps, final output.
3. Click “Add to dataset” (or the equivalent action in the UI).

LangSmith will:

• Take the trace’s input(s) as the dataset example’s “inputs”.
• Optionally store the current output as a “reference” (ground-truth) output if you consider it correct.
• Attach metadata and tags from the trace so you can filter and group later.

You can:

• Add the trace to an existing dataset (e.g., “Production: Support Assistant – Core Scenarios”).
• Or create a new dataset on the fly (e.g., “Checkout Flow Edge Cases – v1”).

This one-click path is the core mechanism: production trace → dataset example.

4. Build richer datasets from multiple traces

Over time, you should expand each dataset into a regression suite that covers:

• Core workflows: e.g., “Book shipment”, “Summarize complaint”, “Check policy eligibility”.
• Known tricky cases: ambiguous questions, long documents, multi-hop reasoning.
• Policy-sensitive flows: actions that touch money, PII, or irreversible changes.

You can do this in several ways:

• Manual curation: Keep browsing traces and adding examples.
• Sampling by filters: Use filters (model version, tool usage, latency) to find interesting subsets, then batch-add selected traces to a dataset.
• From Insights Agent patterns: When Insights surfaces a persistent pattern or failure cluster, drill in and add representative traces to a dedicated dataset.

Each dataset example includes:

• Input fields (often just input, but can be multiple fields for structured calls).
• Optional expected output or rubric (e.g., “must include these facts”, “must not mention X”).
• Metadata (tags for difficulty, domain, priority).

You are effectively turning production reality into a labeled test set.

5. Label and refine with human-in-the-loop

If you’re using LLM-as-judge evaluators, you’ll quickly realize that calibrating them is key. LangSmith helps you inject human judgment directly into the datasets:

• Annotation Queues: Route examples from your dataset into an annotation queue for subject-matter experts.
  • Ask them to rate answer quality, note missing information, or mark policy violations.
  • Collect structured labels (scores, categories) and freeform comments.
• Align Evals: Use these human corrections and few-shot examples to refine your LLM-as-judge evaluators so their scores align with what your experts consider “good”.

Typical labels you might collect:

• Pass / fail or a 1–5 quality score
• Specific error categories (hallucination, missed requirement, tone violation, unsafe action)
• Policy flags (PII handling, compliance, security issues)

These labels become the ground truth for regression testing. Over time, you rely on LLM-as-judge for scale, but you always have human-labeled examples anchoring the metric.

6. Attach evaluators for regression testing

With a dataset in place, you define evaluators that will run on each example when you test a new version of your agent.

Common evaluator types:

• Exact / semantic match: Compare the current output to a reference answer (string match, embedding similarity).
• LLM-as-judge: Ask a model to score the output against a rubric (correctness, completeness, tone).
• Policy / safety checks: Look for PII, banned topics, or disallowed actions.
• Format / structure checks: Ensure output adheres to schemas (JSON, Markdown sections, function call shapes).

In LangSmith, you can:

• Register evaluators programmatically (Python/TypeScript SDKs).
• Or use built-in templates for common evaluation patterns.

You then associate those evaluators with:

• A given dataset (for offline/routine regression).
• Or an experiment run that compares multiple versions.

7. Run the dataset as a regression suite

When you change prompts, models, tools, or orchestration logic, you want to answer two questions:

1. Did we make anything worse on known scenarios?
2. Did we improve the failure modes we targeted?

LangSmith’s offline evaluation workflow:

1. Select the dataset you want to use as your regression suite (e.g., “Support – Top 200 Real Tickets”).
2. Run your new agent version against the dataset:
   • Point to a given model/prompt chain, LangGraph flow, or agent endpoint.
   • Execute all dataset examples, capturing new traces.
3. Apply evaluators to the new outputs:
   • Compute scores and labels based on your metrics (accuracy, policy, format).
4. Compare against previous runs:
   • Side-by-side view of old vs new outputs.
   • Diff of evaluator scores (e.g., “accuracy +7%, safety violations –40%, formatting failures +2%”).

This regression suite behaves like a traditional test suite, but built on real, multi-step agent behavior rather than synthetic unit tests.

8. Close the loop: use production runs to grow the regression set

The flywheel that makes this powerful:

1. Observe production:
   • Traces show you exactly what the agent did and why.
   • Online evals and user feedback highlight failures.
2. Capture failures and key flows as datasets:
   • One-click from trace to dataset example.
3. Evaluate offline before shipping changes:
   • Run datasets against new versions.
   • Catch regressions using evaluators tuned with human feedback.
4. Deploy with confidence on a durable runtime:
   • Use LangSmith Deployment for long-running, stateful agents with checkpointing, exactly-once execution, and rollbacks.
5. Repeat:
   • As new issues show up in production, add them back into your datasets.

Over time, your datasets become a living record of what matters in your domain, and your regression tests get better every week.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Trace capture from any stack	Records full execution trees (inputs, tool calls, model responses, metadata) from production.	You can see exactly what happened, in what order, and reuse those runs as test cases.
One-click “Add to dataset” from traces	Converts a production run into a dataset example, with inputs, outputs, and metadata preserved.	Turns real traffic and failures into regression tests without extra scripting.
Annotation Queues & evaluators	Lets experts label examples and calibrate LLM-as-judge evaluators attached to datasets.	Your regression metrics reflect human quality standards, not just token counts.

Ideal Use Cases

• Best for teams hardening a production agent: Because it lets you capture real failure cases and reuse them as a standing regression suite every time you tweak prompts, tools, or models.
• Best for teams without an initial labeled dataset: Because you can start by capturing production traces, then gradually curate and label the highest-impact flows directly in LangSmith.

Limitations & Considerations

• You still have to curate: LangSmith makes it easy to turn traces into datasets, but you still need to decide what “good” looks like and which runs matter most. Use tags, metadata, and annotation to keep datasets focused.
• Evaluators need calibration: Raw LLM-as-judge scores can drift. Anchor them with human-labeled examples via Annotation Queues and Align Evals so your regression numbers stay meaningful.

Pricing & Plans

LangSmith is designed for teams of any size, with usage-based pricing on traces and datasets plus seat-based access for collaborators.

Typical structure:

• Free or starter tiers to get tracing and basic evals running.

• Paid tiers that expand:

  • Trace volume and retention (e.g., from short-term to extended—hundreds of days).
  • Dataset and evaluator scale.
  • Deployment options (US/EU data residency, hybrid, self-hosted).
  • Enterprise controls (SSO/SAML, SCIM, RBAC/ABAC, audit logs).

• Team / Growth Plans: Best for product and engineering teams needing multi-user access, higher trace volume, and robust regression testing across multiple services.

• Enterprise Plans: Best for large organizations needing longer retention, VPC or self-hosted options, compliance, and fine-grained admin controls around who can view traces, datasets, and eval results.

For current details and a plan fit for your volume and governance needs, contact LangChain directly.

Frequently Asked Questions

Do I need to use LangChain or LangGraph to create datasets from traces?

Short Answer: No. LangSmith is framework-agnostic.

Details: You can instrument any agent stack—OpenAI or Anthropic SDKs, custom Python/TS services, or other frameworks—using the LangSmith SDKs or OpenTelemetry. Once traces are flowing, the dataset workflow is the same: open a run, inspect it, and add it to a dataset. You don’t need to adopt LangChain or LangGraph to benefit from trace-based datasets and regression testing.

What if I don’t have any labeled data yet?

Short Answer: Start with production traces, then layer labels on top.

Details: You capture production traces with LangSmith, then sample interesting or problematic runs into a dataset. Use LLM-as-judge evaluators to bootstrap initial labels, and route high-value examples into Annotation Queues for human review. Over time, you get a growing set of human-aligned labels and evaluators that you can trust for regression tests—even though you started with zero pre-existing data.

Summary

Creating a dataset in LangSmith from production traces is straightforward: you instrument your agent for tracing, watch how it behaves in the real world, and one-click convert key runs into dataset examples. You then use Annotation Queues and evaluators to turn those examples into a regression suite that runs every time you change prompts, models, or logic. Because the data comes from real traffic—not synthetic edge cases—you catch the regressions that actually matter to users.

Next Step

Get Started