We had an agent incident and can’t explain why it took certain actions—how do teams do postmortems for multi-step agents?

Most teams only discover how little they understand their agents when something goes wrong. A user gets a dangerous suggestion, a workflow loops until rate limits trip, or a tool gets called with the wrong parameters—and nobody can reconstruct why. Postmortems for multi-step agents look very different from classic web or microservice incidents; they require full-fidelity traces, per-step evaluations, and a repeatable way to replay and probe the agent’s path.

Quick Answer: To run effective postmortems for multi-step agents, you need full trace-level visibility (spans for every tool call and model hop), structured evaluations on key steps, and a repeatable workflow: capture the failing trace, reconstruct the context and path, layer on evals, then feed what you learned back into datasets, experiments, and guardrails. Teams that rely only on logs or metrics—without traces and evals—end up guessing why the agent did what it did.

Why This Matters

Multi-step agents introduce a new class of production risk: “it sort of made sense along the way, then failed in a way nobody can explain.” Unlike a single LLM call, agents branch, loop, and call tools—often in ways that look wrong in isolation but still converge to a decent answer. When incidents happen, you need to see not just the bad output, but the full conversation, tool calls, intermediate reasoning, and routing decisions that led there.

If you can’t explain what happened, you can’t:

• Prove to security, compliance, or customers that you’re in control.
• Patch the real failure mode instead of just tweaking prompts.
• Turn incidents into reusable test cases that prevent regressions.

Key Benefits:

• Faster incident triage: Span-level traces show you exactly where the agent diverged—no more hunting through scattered logs or ad-hoc console prints.
• Better, repeatable fixes: Evaluations and datasets turn a one-off failure into a regression test you can use to validate prompt, model, and router changes.
• Stronger governance: End-to-end visibility and auditable histories give risk and compliance teams a concrete view of what your agents actually did.

Core Concepts & Key Points

Concept	Definition	Why it's important
Traces & spans for agents	A trace is the full execution of a user session; spans are the individual steps—model calls, tools, routers, sub-agents—captured via OpenTelemetry/OpenInference.	Without spans, multi-step agents are black boxes; you can’t reconstruct why they made a decision or where a path went wrong.
Path & step evaluations	Structured checks—LLM-as-a-Judge, code evals, heuristics—applied to specific steps (tool choice, parameter extraction, safety, path length).	They let you quantify “good vs. bad” behavior per step and detect the exact moment an incident started, instead of just seeing the final failure.
Incident-to-dataset loop	Turning every significant incident trace into labeled data, tests, and CI experiments.	This closes the loop between production and development so your agent gets measurably more robust over time instead of reintroducing the same bugs.

How It Works (Step-by-Step)

One platform. Postmortems that actually improve your agents.

At my day job, our rule is simple: we don’t ship or scale an agent unless we can trace every step, evaluate every critical sub-call, and wire incidents back into experiments. Here’s the exact pattern we use with Arize AX (and Phoenix if you prefer self-hosted open source) to do post-incident analysis on multi-step agents.

1. Capture the incident trace with full context

   The first question in any postmortem is: “Can we see the full flow, or are we reconstructing from scraps?” That’s where OpenTelemetry-based tracing pays off.

   Concretely:

   • Instrument your agent so every model call, tool call, router decision, and sub-agent execution creates a span.
   • Use OpenInference-style conventions so spans are consistent regardless of framework (LangGraph, ReAct-style, custom orchestration, etc.).
   • Ensure you log:
     • Full prompts and responses (or redacted versions for PHI/PII).
     • Tool inputs/outputs, including validation errors or retries.
     • Routing decisions and scores (e.g., which tool candidate was chosen and why).
     • Session metadata: user ID (hashed), tenant, channel, feature flag version, model version, prompt version, retrieval corpus, etc.

   When an incident happens, the first step in the postmortem is simply: pull up the trace for that specific session in Arize AX or Phoenix. You should see a multi-agent graph (or tree) that lets you click into each span and inspect the inputs/outputs.

2. Reconstruct the “why” with path-level inspection

   Once you have the trace, you’re not just staring at a bad output—you’re walking the path that produced it.

   For each step:

   • Identify the first wrong turn.
     • Did the router pick the wrong tool or sub-agent?
     • Did retrieval pull irrelevant documents?
     • Did the model hallucinate a tool result instead of waiting for the tool?
   • Check the working memory.
     • What context was carried across steps? Did the agent lose key constraints (like jurisdiction, budget, or data scope) along the way?
     • Did a step overwrite or drop important state?
   • Observe path shape.
     • Did the agent loop (e.g., back to the router) more than expected?
     • Did it take a long path with redundant tools or unnecessary hops?

   Early in development, I always recommend manual trace inspection for incidents. Humans are still much better at pattern-spotting complex path errors—like an agent that “kind of” works but takes five extra steps first—than any automated metric.

3. Layer evaluations onto critical steps

   Once you’ve found the failing behavior, you want to make it measurable. This is where step-level evaluations come in.

   In practice, we define evaluators for:

   • Tool selection accuracy
     • LLM-as-a-Judge: “Given the user query and available tools, did the agent choose the correct tool(s)?” Score on a 1–5 scale with reason.
     • Use this to catch incidents where the agent keeps querying a read-only tool instead of the write-capable one, or hits the wrong data source.
   • Parameter extraction & schema adherence
     • Code evals: Validate that the JSON or arguments passed to a tool conform to your schema and constraints.
     • LLM-as-a-Judge: “Are these parameters consistent with the user’s request and policies?”
   • Safety & policy compliance
     • LLM-as-a-Judge: Check for policy violations (PHI leakage, disallowed actions, unsupported jurisdictions).
     • Add both pre-call (should we even do this?) and post-call (did we respect the constraints?) checks.
   • Path health
     • Simple heuristics: number of steps, number of router hops, number of repeated tool calls.
     • These “iteration counters” are crucial to catch path explosions and loops that don’t technically fail but kill latency or quota.

   With Arize AX, you can run these as offline evals across the incident trace and similar traces, or as online evals to catch problems in real time. Phoenix gives you a self-hosted way to run similar evaluations and inspections.

4. Turn the incident into a permanent test case

   A postmortem isn’t done when you understand the incident; it’s done when you can prove you’ll catch it next time.

   For every serious incident:

   • Extract a dataset row from the failing trace:
     • Inputs: user request, environment metadata, relevant context (e.g., retrieved docs).
     • Expected behavior: description or canonical “right” outcome (which tools to call, which constraints to honor).
   • Label it in an annotation queue:
     • Use human annotators (often your own engineers or domain experts) to:
       • Mark the correct tool selection.
       • Provide the correct parameters or answer.
       • Highlight where the path diverged.
   • Store it as a golden dataset in Arize AX:
     • Tag it with the incident ID, root cause, and severity.
     • Include nearby “hard negatives” (similar prompts that should be treated differently) to tighten your agent’s discriminative behavior.

   Now, the incident is not just a story in a doc; it’s a structured example that your training, fine-tuning, retrieval configurations, or prompt templates can optimize against.

5. Run experiments to validate and gate fixes

   Once you’ve proposed a fix—new prompt, different router logic, an added safety check—you want to run an experiment, not just ship and pray.

   With Arize AX Experiments:

   • Create an experiment that compares:
     • Old vs. new prompt.
     • Old vs. new tool router.
     • Old vs. new model or retrieval config.
   • Use your incident-enriched dataset as part of the evaluation set:
     • Ensure the specific incident case and similar edge cases are included.
     • Add your evaluators: tool selection, safety, parameter correctness, path length.
   • Define pass/fail gates:
     • Example: “New variant must improve tool selection accuracy on incident-related cases AND not regress safety or latency on the broader dataset.”
   • Wire into CI/CD:
     • Make these experiments required checks for any change that touches prompts, routers, or critical tools.
     • If the experiment fails, the change doesn’t ship.

   This is how we make sure that “fixing” one agent incident doesn’t quietly introduce five new ones somewhere else.

6. Monitor for recurrences with online evals and alerts

   Postmortems don’t help if incidents silently return. After you’ve added tests and shipped a fix, set up monitoring:

   • Online evals in Arize AX:
     • Run your key LLM-as-a-Judge and code evaluators on live traffic (or a sampled subset).
     • Track metrics like tool selection accuracy, safety score, parameter validity, and path length over time.
   • Dashboards & alerts:
     • Create dashboards focused on agent SLOs:
       • Response quality (eval scores).
       • Path health (median/max steps, loop indicators).
       • Safety violations.
       • Cost and token usage per session.
     • Set alerts on:
       • Sudden drops in eval scores.
       • Spikes in path length, retries, or timeouts.
       • Increases in flagged safety violations or policy breaches.

   When something regresses, you get notified with direct links to recent traces and can kick off the same postmortem workflow again—faster each time because the pattern is standardized.

Common Mistakes to Avoid

• Relying on logs instead of traces:
  Raw logs and print statements don’t capture multi-agent flows. Without OTEL-based spans, you’ll miss cross-service context, tool responses, and routing decisions. Instrument once with open standards so every agent run is traceable across models, tools, and services.

• Only evaluating final answers, not the path:
  Many teams score just the final answer quality. For multi-step agents, that hides the real problem: incorrect tool calls that happen to cancel out, unsafe intermediate steps, or path explosions. Evaluate specific sub-steps—tool choice, parameter extraction, safety checks, and path length—so you catch problems even when the output “looks fine.”

Real-World Example

At our marketplace, we had an incident where an internal support agent tried to modify account records it wasn’t supposed to touch. The final answer looked okay—it politely declined the user’s request—but behind the scenes, the agent made several unauthorized write attempts that failed at the API layer. Security caught it in logs; engineering couldn’t easily explain what happened.

Here’s how we used a structured postmortem to fix it:

1. Trace reconstruction:
   Using Arize OpenTelemetry tracing, we pulled the full multi-agent trace. We saw that:

   • The router correctly picked the “Account Service” tool.
   • The agent repeatedly attempted writes with missing authorization context.
   • A safety sub-agent checked the user request but not the downstream tool arguments.

2. Root cause:
   The agent’s working memory had the user’s role, but the tool schema didn’t require it, so the agent guessed. Our safety checks were scoped to the natural-language request, not the actual tool payloads.

3. Evaluations:
   We added:

   • A parameter correctness eval to assert that every write tool must include explicit authorization context.
   • A safety eval that reviewed tool arguments, not just the original user prompt.

4. Dataset & experiment:
   We turned the incident into a labeled dataset row:

   • “User with role X requests Y; agent must either decline or route through the ‘Access Review’ sub-agent.” We then ran an Arize AX experiment comparing:
   • Old vs. new tool schema + prompts + safety guardrails. The new variant had:
   • 100% compliance on the incident and similar cases.
   • No regression on other workflows.

5. Monitoring:
   Finally, we set online evals to watch for:

   • Any write tool call without an explicit authorization span.
   • Any safety eval below a threshold on write operations.

What started as a scary “why did it try to modify that record?” incident turned into a hard guardrail that now protects every downstream agent we build.

Pro Tip: Don’t wait for a big incident to define your incident playbook. Take a mild but interesting failure, run a full postmortem on it, then templatize that workflow (trace → eval → dataset → experiment → monitor). When a real incident hits, you’ll already have the rails in place.

Summary

Postmortems for multi-step agents can’t stop at “we’ll tweak the prompt and monitor.” They need full trace visibility, per-step evaluations, and a disciplined loop that turns every incident into data, tests, and CI/CD gates. When you can replay traces, inspect tools and routers, score each step, and automatically test fixes against real-world failures, you move from demoing agents to shipping agents that work—and stay working as you iterate.

Next Step

Get Started