Our agent fails in production but we can’t reproduce it—how do we debug a non-deterministic, multi-step workflow end-to-end?

Most teams hit this wall the first time they ship a serious agent: it fails in production, the user is unhappy, and you can’t get it to fail the same way again in staging. With long context windows, branching tool calls, and non-deterministic LLM outputs, you’re no longer debugging a stack trace—you’re debugging a 200-step reasoning trajectory that only existed once, at runtime.

Quick Answer: You debug non-deterministic, multi-step agents by capturing full production traces (not just logs), replaying the exact run timeline, and turning those traces into datasets and evals. LangSmith gives you an end-to-end workflow to observe, reproduce, evaluate, and fix failures even when you can’t naturally repro them in dev.

---

The Quick Overview

• What It Is: An end-to-end, trace-first workflow in LangSmith for debugging non-deterministic, multi-step AI agents: capture runs, inspect decisions, replay behavior, and promote fixes with evals and rollbacks.
• Who It Is For: Teams building production agents and complex LLM workflows—support copilots, research agents, ops automation, internal tools—who see “we can’t reproduce it” failures.
• Core Problem Solved: Traditional logs don’t explain why an agent went off the rails, and flaky, non-deterministic behavior can’t be reliably repro’d. You need full traces, structured timelines, and evaluation loops, not just console logs and manual spot checks.

---

How It Works

The core idea: treat every production failure as a trace you can replay, not a bug you hope to reproduce.

LangSmith instruments your agent so every execution becomes a structured trace composed of runs (LLM calls, tool invocations, chains, sub-agents). That trace is the exact record of what happened in production. From there, you can:

• Inspect the full multi-step timeline.
• See prompts, context, tool calls, and intermediate results.
• Extract the failing slice into a dataset.
• Run offline evals (LLM-as-judge calibrated with humans) before shipping fixes.
• Deploy updated agents with durable runtime guarantees and rollbacks.

End-to-end, the debugging lifecycle looks like this:

1. Observe: Capture and inspect the failing trace

   • Every agent run in production emits a trace into LangSmith: messages, tool calls, inputs/outputs, latencies, and errors.
   • When a user reports a failure, you find the exact production trace and open a run timeline that shows what happened, in what order, and with what context.
   • You identify where the agent went off track: wrong tool choice, missing context, hallucinated answer, loop, or policy violation.

2. Evaluate: Turn the failure into a test and iterate

   • You one-click convert the failure trace (and similar runs) into a dataset.
   • You attach evaluators—LLM-as-judge, custom functions, or human annotations via queues—to score behavior across that dataset.
   • You adjust prompts, tools, routing, or memory, then compare before/after behavior on the same failing examples to make sure the fix actually addresses the real-world issue and doesn’t introduce regressions.

3. Deploy: Ship the fix with guardrails and rollbacks

   • You deploy the updated agent onto LangSmith’s runtime (or keep your own stack instrumented) with exactly-once execution, durable checkpointing, and versioning.
   • You run online evals on live traffic and monitor metrics, with the ability to roll back to a previous agent configuration if things regress.

---

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Trace-first run timelines	Captures every LLM call, tool invocation, and chain step as structured runs inside a trace.	Lets you replay a non-deterministic, multi-step workflow exactly as it happened in production.
Trace → dataset conversion	Turns real production traces (including failures) into reusable evaluation datasets.	Makes “can’t repro” bugs testable offline and keeps your eval suite grounded in real user behavior.
Calibrated evals & comparison	Runs LLM-as-judge, heuristics, and human labels; compares agent versions side-by-side.	Lets you validate fixes on real failure cases and catch regressions before pushing to production.
Annotation queues & Align Evals	Routes traces to experts for labeling; uses labels to tune evaluators and prompts.	Aligns automated evals with human judgment so you can trust scores for shipping decisions.
Durable agent deployment	Provides memory, threads, checkpointing, exactly-once execution, and version control.	Keeps long-running agents stable and debuggable, with rollbacks when non-deterministic behavior regresses.
Framework-agnostic instrumentation	Integrates via LangChain, LangGraph, Deep Agents, OpenTelemetry, or SDKs (Python, TS, Go, Java).	Works with any agent stack and model provider without locking you into a specific framework.

---

How to Debug a Non-Deterministic, Multi-Step Workflow in Practice

1. Instrument reality: capture full traces, not just logs

You can’t fix what you can’t replay.

With agents, “the bug” isn’t a null pointer at line 57; it’s “why did the agent call edit_file instead of read_file at step 23 of 200?” That’s a reasoning failure, not a code failure.

To debug that, you need:

• Runs: Each LLM call, tool invocation, chain, or sub-agent step as a typed unit (inputs, outputs, metadata).
• Traces: A tree/graph of runs representing the full agent trajectory for a user request.
• Context: Prompts, system messages, retrieved documents, tool descriptions, and intermediate state.

In LangSmith, you:

• Add the SDK (Python, TypeScript, Go, or Java) or enable native tracing for LangChain, LangGraph, Deep Agents, or OpenTelemetry.
• Wrap your agent entrypoint with tracing so every production request produces a trace.
• Attach metadata: user IDs, tenant, environment, version, experiment flags, etc. for filtering later.

Now, when a failure occurs in production, you have a complete record of the agent’s reasoning path.

2. Find the failure trace and pinpoint the divergence

When someone says “the agent gave a wrong answer” or “it executed the wrong tool,” you’re usually missing:

• The exact prompt and context at that time.
• The full interaction history leading up to it.
• The chain of tool calls and decisions that set it up to fail.

With LangSmith:

1. Locate the trace

   • Filter by:
     • User, session, environment (prod/staging).
     • Error flags, low evaluation scores, or manual error tags.
     • Time window around when the issue was reported.
   • Open the trace in the run viewer.

2. Inspect the run timeline

   • Follow the call tree: high-level agent run → child runs (tools, sub-agents, retrieval, LLM calls).
   • For each LLM call, see:
     • System and user prompts.
     • Few-shot examples and tool descriptions.
     • Retrieved documents or context snippets.
   • For each tool call, see:
     • Inputs the agent constructed.
     • Outputs returned.
     • Latency and any errors.

3. Identify the failure type Common non-deterministic failures are:

   • Bad tool choice: Called the wrong tool (delete_order vs fetch_order).
   • Missing or wrong context: Retrieval missed critical facts; context window truncated important messages.
   • Hallucination: Confident but incorrect answer not grounded in any retrieved source.
   • Policy violation: The agent answered something it shouldn’t, or used non-approved language/actions.
   • Control-flow bugs: Looping, excessive tool calls, or failing to terminate.

The trace lets you answer, step-by-step:

• What context and instructions existed right before it made the wrong decision?
• Where in this 200-step trajectory did the agent go off track?
• Were there earlier “almost failures” that you’re not yet monitoring for?

3. Extract the failing behavior into a dataset

You don’t want to fix this once and hope it doesn’t show up again. You want this exact failure—and others like it—to become part of your permanent test suite.

In LangSmith:

1. Create a dataset from trace(s)

   • Select the failing trace.
   • Extract the relevant state:
     • User input(s) that led to the failure.
     • Any necessary context (prior turns, metadata).
   • Save this as a dataset entry (or multiple entries) tagged with the failure type.

2. Enrich with labels if you know the correct behavior

   • Add:
     • Expected output (ground truth).
     • “Success/failure” flags and failure category (hallucination, routing, retrieval, safety, etc.).
   • If you don’t know the ideal final answer, you can still label relative behavior (e.g., “tool choice wrong,” “unsafe content”).

Over time, production traces become your most valuable evaluation asset:

• You’re not inventing synthetic test cases.
• Users effectively “write” your evals by triggering real edge cases.
• Your dataset is always representative of actual workloads.

4. Attach evaluators: make the bug measurable

To avoid debugging by vibes, you need metrics.

LangSmith supports:

• LLM-as-judge evaluators
  • Ask a model to grade agent outputs: correctness, relevance, tone, safety, tool usage.
  • Use Align Evals to calibrate these evaluators with human corrections and examples so scores track your actual quality bar.
• Rule-based/heuristic evaluators
  • Custom functions that assert invariants:
    • “No PII leakage.”
    • “Always call get_customer_profile before update_customer_profile.”
    • “Never execute transfer_funds without an approval flag.”
• Human evaluations
  • Route dataset entries into an annotation queue.
  • Let SMEs label “good/bad” and provide corrections or policy judgments.

Attach these evaluators to your dataset so you can quantify:

• How often the old agent fails this scenario.
• Which failure modes are most common (routing vs retrieval vs safety).
• How sensitive behavior is to prompt or configuration changes.

5. Iterate on prompts, tools, and flows with side-by-side comparisons

Now you fix the agent—but you do it with a feedback loop.

Possible interventions:

• Prompt changes: Clarify when to use which tools, how to ground answers, or how to ask for clarification.
• Tooling changes: Add missing tools, tighten tool schemas, or enforce safer defaults.
• Routing logic: Adjust policies for when to call certain sub-agents or external systems.
• Memory and context: Change how history is summarized, what gets retrieved, or how long the conversation thread persists.

In LangSmith:

1. Create a new agent version (new prompt, new routing, or new code).
2. Run the same dataset through both:
   • Baseline: current production behavior.
   • Candidate: modified behavior.
3. Compare:
   • Overall evaluation scores (LLM-as-judge + rules + humans).
   • Per-case diffs: see where the candidate fixed the failure and where it got worse.
   • Cost, latency, tool usage patterns.

This comparison step is where non-determinism stops being scary. You’re not asking “does it feel better?” You’re asking:

• Does the candidate agent fix the original failure?
• Does it break any previously good behavior?
• Is there any new, unacceptable failure mode introduced by the change?

Only when you’re satisfied with these answers do you move on to deployment.

6. Deploy with a durable runtime and rollback plan

Non-deterministic agents don’t just fail in one-shot transactions; they fail in long-running, multi-turn workflows. If you can’t replay and roll back, you’re flying blind.

LangSmith’s deployment layer is built for this:

• Memory and threads
  • Maintain conversation threads and task state across steps.
  • See the full history that influenced each decision.
• Durable checkpointing
  • Persist agent state between tool calls, external callbacks, or long-running operations.
  • Recover and inspect mid-run state when something goes wrong.
• Exactly-once execution
  • Prevent double-execution of critical tools in the presence of retries or network issues.
• Versioning and rollbacks
  • Deploy new agent versions.
  • Gradually shift traffic via canary or A/B setups.
  • Roll back fast if online evals or production traces show regressions.

You can also connect Fleet/Agent Builder for non-technical teams to author agents in plain language, with:

• Tool-level approvals (“ask permission before taking sensitive actions”).
• Admin controls and audit logs for who changed what, and when.

All of this is on top of enterprise controls—SSO/SAML, SCIM, RBAC/ABAC, audit logs, US/EU data residency, hybrid and self-hosted options—and a clear data posture: LangSmith does not use your data to train models.

---

Ideal Use Cases

• Best for teams whose agents pass QA but fail in real user traffic: Because production traces become your truth source. You see exactly what happened in the wild and immediately convert that into tests and evals.
• Best for long-running, multi-agent workflows with many tools: Because a simple log line per request can’t explain why a 200-step, 2-minute trajectory went off track—but a trace with structured runs and checkpoints can.

---

Limitations & Considerations

• You need to instrument upfront: If you’re not tracing runs before the failure happens, you can’t retroactively reconstruct the exact reasoning path. Start tracing early—even in staging—so that when prod issues appear, you already have the data.
• Non-determinism never disappears entirely: Even with tracing, datasets, and evals, LLMs will remain probabilistic. The goal is to bound behavior with measurement, guardrails, and rollbacks—not to pretend agents are traditional deterministic code.

---

Pricing & Plans

LangSmith is designed for teams of any size, with usage-based pricing so you pay for the traces and storage you actually use.

Typical structure:

• Team / self-serve plans

  • Seat-based pricing plus pay-as-you-go usage (traces, eval runs, storage).
  • Good for small to mid-size teams getting serious about agent observability and evaluation.
  • Includes core tracing, datasets, evals, and basic retention (e.g., 14-day trace retention with options to extend).

• Enterprise plans

  • Best for larger organizations or regulated environments that need extended retention (up to 400 days), custom SLAs, and advanced deployment options.
  • Includes:
    • SSO/SAML, SCIM, RBAC/ABAC, audit logs.
    • US/EU data residency, hybrid or self-hosted deployments (keep data in your VPC).
    • Higher event volume support (we handle >1B events per day today).
  • Designed for teams running mission-critical agents at scale—for example, Fortune 500 companies automating thousands of orders per day with human-in-the-loop.

For exact pricing, usage tiers, and deployment options, you’ll want to speak with our team.

• Growth / Team: Best for product teams and startups needing robust tracing, evals, and moderate retention to get agents into production.
• Enterprise: Best for large organizations needing long-term trace storage, strict governance, custom deployments, and higher-volume event ingestion.

---

Frequently Asked Questions

How do I debug a failure I can’t reproduce locally?

Short Answer: You don’t try to recreate it from scratch—you pull up the production trace in LangSmith and replay the exact run timeline.

Details: Once your agent is instrumented, every production request creates a trace with:

• Full conversation history and prompts.
• Tool calls and their inputs/outputs.
• Intermediate LLM calls and their context.

When a user reports “it failed,” you:

1. Filter for their user/session/time to locate the trace.
2. Inspect the run timeline to find the step where behavior diverged.
3. Convert that trace (or slice of it) into a dataset entry.
4. Use that dataset to evaluate and iterate on fixes offline.

You never have to rely on “I think it involved these inputs…” again; the trace is the ground truth.

---

How do I know my fix doesn’t introduce new regressions?

Short Answer: Run both the old and new agents against the same trace-derived datasets and compare evaluation scores side-by-side.

Details: With LangSmith:

1. Build a dataset from:
   • Known failures (like the one you just debugged).
   • Representative production traces (successes and borderline cases).
2. Attach evaluators: LLM-as-judge, rules, and human labels.
3. Run:
   • Baseline (current prod agent).
   • Candidate (new prompt/code/routing).
4. Use side-by-side comparison views to check:
   • Which examples improved.
   • Which stayed the same.
   • Which got worse.

Only promote the candidate when metrics and human judgment say it’s better overall—even if the failure that triggered the work is fixed. This is how teams like Klarna, Podium, and C.H. Robinson keep agents improving while avoiding silent regressions.

---

Summary

Non-deterministic, multi-step agents don’t fail like traditional software. There’s no single line of code to fix and no guarantee you can reproduce the problem in dev. The only reliable way to debug is to record reality: capture full production traces, replay the exact run timeline, and turn those traces into datasets and evals that gate what you ship.

LangSmith is built around that premise:

• Trace-first observability so you see exactly what your agent did, in what order, and why.
• Traces → datasets → evals so every production failure becomes a permanent test.
• Durable, governed deployment so you can ship, monitor, and roll back agent versions with exactly-once execution and enterprise controls.

Instead of “our agent fails in production and we can’t reproduce it,” you get “we see exactly where it failed, we turned it into a test, and the next version passes that test before it ever hits users.”

---

Next Step

Get Started