LLM observability platform that works with LangChain + LlamaIndex + custom agents and supports prompt replay/debugging

Most teams don’t realize they need a real LLM observability platform until the first production incident hits: a LangChain agent silently takes a bizarre tool path, a LlamaIndex query goes off the rails, and a custom agent “works on my laptop” but fails under production rate limits—with no trace to debug what happened. If you’re stitching together LangChain, LlamaIndex, and bespoke agents, you need more than logs; you need end-to-end traces, prompt replay, and evaluation-driven iteration anchored on open standards.

Quick Answer: The most reliable way to observe LangChain, LlamaIndex, and custom agents in one place is to standardize on OpenTelemetry-based tracing and plug everything into an LLM observability platform that supports spans, traces, and prompt replay. Arize AX + Phoenix does exactly this: it traces your entire agent flow (framework-agnostic), lets you replay and debug prompts, and layers in evaluations and CI/CD so you can ship prompt and agent changes with confidence.

Why This Matters

When you combine multiple frameworks—LangChain for orchestration, LlamaIndex for retrieval, custom Python agents for business logic—the “happy path” looks clean in diagrams, but production behavior is non-deterministic. Agents follow strange paths, tools fail, and prompts evolve weekly. Without unified observability, you’re flying blind:

• You can’t see which sub-call or tool caused a bad answer.
• You can’t reproduce the exact prompt + context that led to a hallucination.
• You can’t tell whether a new prompt, router, or index actually improved things or quietly regressed quality.

An LLM observability platform built on open tracing and evaluations closes this loop. It lets you trace every LangChain, LlamaIndex, and custom agent call, replay prompts, run evals on top, and gate deployments using CI/CD experiments instead of intuition.

Key Benefits:

• Unified tracing across frameworks: See LangChain chains, LlamaIndex queries, and custom agents as a single trace, not three disconnected logs.
• Prompt replay and debugging: Reproduce any production issue by replaying the exact prompt, context, and parameters in a playground.
• Evaluation-driven CI/CD: Use LLM-as-a-Judge, code evals, and human annotations to detect regressions early before they hit SLOs.

Core Concepts & Key Points

Concept	Definition	Why it's important
Open Standard Tracing (OTEL + OpenInference)	Instrumenting your LangChain, LlamaIndex, and custom agents with OpenTelemetry spans that follow OpenInference conventions for LLM calls, tools, and retrieval.	Gives you framework-agnostic, vendor-neutral traces so you’re not locked into proprietary SDKs or tied to a single agent framework.
Prompt Replay & Debugging	The ability to select any span/trace, view the prompt, model, retrieval context, and tool calls, and rerun or tweak the prompt in a playground.	Turns production incidents into reproducible experiments, so you can debug failures, fix prompts, and verify improvements before redeploying.
Evaluation-Driven CI/CD	Running offline and online evals (LLM-as-a-Judge, code checks, human annotations) on traces and datasets, then using experiments to gate prompt and agent changes.	Replaces “ship and hope” with measurable quality checks, catching regressions in tool selection, parameter extraction, and answer quality before customers see them.

How It Works (Step-by-Step)

At my current company, we standardized on Arize AX (with Phoenix) as the LLM observability layer for LangChain, LlamaIndex, and our custom agents. The pattern is straightforward: trace everything with OTEL, send spans to Arize, then build eval-driven loops on top.

1. Instrument LangChain, LlamaIndex, and custom agents with OTEL spans

   Start by making your agent stack traceable:

   • LangChain:
     • Use OpenTelemetry middleware or LangChain tracing integrations to wrap chains, tools, and LLM calls.
     • Emit spans for key operations: chain calls, tool invocations, retrievers, and intermediate steps.
   • LlamaIndex:
     • Instrument query engines, retrievers, and index builds with OTEL spans.
     • Log query text, node IDs, and similarity scores as span attributes (while respecting PII constraints).
   • Custom agents:
     • Wrap your planner, router, and custom tools in spans.
     • Follow OpenInference conventions so Arize can recognize LLM calls, embeddings, and tool invocations uniformly.

   The goal is a single trace per user request that contains every LLM call, retrieval, and tool step—regardless of framework.

2. Send traces to Arize AX + Phoenix and enable prompt replay

   Once spans are in place:

   • Export OTEL traces to Arize via standard OTEL exporters (HTTP/gRPC).
   • In Arize AX / Phoenix, each request appears as a trace graph with spans for:
     • LangChain chains and tools
     • LlamaIndex retrievals
     • Custom agent decisions and business logic
   • For each LLM/tool span, Arize stores:
     • Prompt + system message
     • Model, temperature, top_p, etc.
     • Retrieval context (documents, nodes, or references)
     • Tool input/output payloads (schemas, parameters, results)
   • Use the Prompt Playground & Management in AX to:
     • Select a production span and replay the prompt with the same or a different model.
     • Tweak prompts, system instructions, and parameters to see how responses change.
     • Promote improved prompts into a managed prompt version if they outperform baselines.

   This makes debugging concrete: “this LangChain agent picked the wrong tool for order refunds” becomes “replay span #XYZ, adjust the decision prompt, validate results across a dataset.”

3. Layer on evaluations and CI/CD experiments to catch regressions

   Traces tell you what happened; evals tell you whether it was good. In Arize AX, we close the loop:

   • Offline Evals:
     • Use Arize’s LLM as a Judge templates to score:
       • Answer correctness and hallucination risk
       • Tool selection quality (did the agent choose the right tool?)
       • Parameter extraction accuracy (did it parse entities and IDs correctly?)
       • Path convergence (did multi-step agents reach a correct final answer?)
     • Add code evals for deterministic checks:
       • JSON schema validity
       • SLA constraints (latency, token counts)
       • Business rules (“never show PII,” “must include a refund policy link”)
   • Human Annotation & Queues:
     • Route ambiguous or high-impact traces into annotation queues.
     • Build golden datasets from real production edge cases: tricky queries, weird tool outputs, domain-specific questions.
   • CI/CD Experiments:
     • Before you change a LangChain prompt, LlamaIndex retriever config, or agent routing logic:
       • Create an experiment in Arize with your golden dataset.
       • Compare old vs. new:
         • Prompts
         • Models
         • Retrieval params (k, filters, rerankers)
         • Agent strategies (tools, paths)
       • Use eval scores and guardrails to gate the release. If hallucination, tool-call failure, or latency increases beyond a threshold, the experiment fails and the change doesn’t ship.
   • Online Evals & Monitoring:
     • Turn on Online Evals to continuously score live traffic.
     • Configure dashboards and alerts to monitor:
       • Answer quality / hallucination rate
       • Tool-call error rates
       • Latency and token usage by agent, model, or framework (LangChain vs. custom)
     • When something drifts, click directly into traces and replay the offending prompts.

This is how you move from demo-mode to “Ship Agents that Work” across LangChain, LlamaIndex, and custom stacks.

Common Mistakes to Avoid

• Treating each framework as a separate “black box”:
  How to avoid it: Don’t rely on LangChain’s built-in logging here, LlamaIndex logs there, and homegrown logs elsewhere. Standardize on OpenTelemetry + OpenInference so every call—LLM, tool, retriever—is a span in the same trace. Then point that firehose at Arize AX or Phoenix so you can see the full path.

• Skipping evaluation in favor of anecdotal testing:
  How to avoid it: Never ship a new prompt, index config, or agent router based on spot checks. Use datasets + experiments with LLM-as-a-Judge, code evals, and annotation queues in Arize. Bake these into CI/CD so merges and releases are blocked when quality regresses.

Real-World Example

At my marketplace, our “Order Help” assistant is a composite system:

• LangChain orchestrates a few high-level tools (order lookup, refund calculator, policy fetcher).
• LlamaIndex serves as the retrieval layer over policy documents and SOPs.
• A custom Python agent handles compliance rules and domain-specific decisions.

We initially logged each component separately. When a surge of “incorrect refund” complaints hit, we had no way to reconstruct what happened: sometimes the LangChain agent picked the wrong tool, other times LlamaIndex retrieved an outdated policy, and occasionally our custom logic overrode the right answer.

We re-architected observability around Arize AX + Phoenix:

1. Instrumented LangChain chains, LlamaIndex queries, and custom tools with OTEL spans, following OpenInference to label LLM/tool/retriever calls.
2. Sent traces into Arize and used multi-agent trace graphs to visualize the entire flow per user session.
3. Turned problematic chats into datasets, then:
   • Ran LLM-as-a-Judge evals for refund correctness and explanation quality.
   • Added code evals for policy compliance (e.g., “refund does not exceed order value,” “no PII leak”).
   • Prioritized ambiguous cases into annotation queues and built a golden dataset from real incidents.
4. Used Prompt Playground & Management to replay failing spans:
   • Tweaked LangChain decision prompts to better distinguish shipping delays vs. damaged items.
   • Adjusted LlamaIndex retrieval parameters and reranker.
   • Verified changes across the golden dataset before promotion.
5. Wrapped all of this in CI/CD experiments:
   • Every change to prompts, retrieval configs, or agent routing now runs through Arize experiments.
   • Releases are automatically blocked if correctness drops or hallucinations increase on key slices (e.g., high-value orders, specific regions).

Within a couple of weeks, “mysterious refund mistakes” dropped sharply. More importantly, any new issue now comes with a full trace we can replay, annotate, and learn from.

Pro Tip: When instrumenting, treat tool calls and retrievers as first-class spans, not just attributes on LLM calls. That’s what unlocks true path debugging—you can see when a LangChain agent chose the wrong tool or when a LlamaIndex query returned irrelevant nodes, then target your fixes precisely.

Summary

If your stack mixes LangChain, LlamaIndex, and custom agents, a generic logging setup won’t cut it. You need:

• Open standard tracing (OTEL + OpenInference) so you can see the full agent path across frameworks.
• Prompt replay and debugging to reproduce production issues and iterate quickly.
• Evaluation-driven CI/CD and monitoring to catch regressions before they break SLOs.

Arize AX, backed by the open-source Phoenix tracer, is built around exactly this loop: trace every span, evaluate every sub-call, and connect production behavior back into experiments and prompt management. That’s how you move from promising demos to reliable, production-grade agents—without locking into proprietary frameworks or sacrificing observability when your stack evolves.

Next Step

Get Started