How do I add observability to a LangChain/LlamaIndex app so I can see each step of the chain per request?

Most teams building with LangChain or LlamaIndex eventually hit the same pain point: once your app leaves the notebook and starts handling real traffic, it becomes very hard to see what’s happening inside each request. You know the input and the final answer, but not the intermediate steps, model calls, tool executions, or where latency and errors actually come from.

This is exactly the problem observability solves. With proper tracing, you can inspect each step of the chain or graph per request, debug quickly, and systematically improve performance and safety.

Below is a practical guide to adding observability to a LangChain or LlamaIndex app, with examples of how to do this using Langtrace—an observability and evaluation platform purpose-built for AI agents and LLM apps.

---

Why observability is critical for LangChain & LlamaIndex apps

Modern AI apps are multi-step pipelines:

• Retrievers, vector stores, and tools
• Chains, graphs, and agents
• Multiple model calls (LLMs, embeddings, rerankers)
• External APIs and databases

Without observability, you’re blind to:

• Which step is slow or failing
• How prompts evolve through the chain
• Whether the right documents are being retrieved
• Where hallucinations or unsafe outputs originate

To improve performance and security, you need both:

1. Observability – tracing every step per request
2. Evaluations – measuring quality, safety, and reliability on those traces

Langtrace is designed to give you both with minimal instrumentation, while supporting the frameworks you already use: LangChain, LlamaIndex, CrewAI, and DSPy, plus a wide range of LLM providers and VectorDBs.

---

Core concepts: what “step-level” observability means

When you add observability to a LangChain or LlamaIndex app, you should be able to:

• See a trace per request – a timeline showing everything that happened
• Inspect each step – prompts, model parameters, responses, errors
• Drill into sub-steps – tool calls, retriever queries, vector store interactions
• Correlate latency and failures – identify slow or broken components
• Replay or analyze – send traces into evaluations or regression tests

Langtrace captures this as a parent trace (the whole request) with child spans (each chain/tool/model step).

---

Step 1: Set up Langtrace for your project

To add observability with Langtrace, start by connecting your app to the Langtrace backend.

1. Create a project and API key

1. Sign up and log in to Langtrace.
2. Create a new project.
3. Generate an API key for that project.

You’ll use this API key in your backend service so Langtrace can receive traces from your LangChain or LlamaIndex app.

2. Install the Langtrace SDK

Choose the SDK that matches your stack (typically Python for LangChain and LlamaIndex applications).

For example, in Python:

pip install langtrace

(Use the exact package name and installation instructions from Langtrace’s docs for your language or framework.)

3. Instantiate Langtrace with your API key

In your app’s initialization/bootstrap code:

from langtrace import Langtrace

lt = Langtrace(api_key="YOUR_LANGTRACE_API_KEY")

You usually do this once at startup (e.g., in app.py, main.py, or your FastAPI/Flask/Django entrypoint).

---

Step 2: Add observability to a LangChain app

LangChain already has a strong concept of “runs” and callbacks. Langtrace hooks into this so you can capture each step automatically.

1. Integrate Langtrace with LangChain callbacks

Langtrace typically provides a callback handler compatible with LangChain’s callback system. After you instantiate Langtrace, you can grab its handler and pass it into your chains.

For example:

from langchain_openai import ChatOpenAI
from langchain.chains import LLMChain
from langchain.prompts import PromptTemplate
from langtrace import Langtrace

lt = Langtrace(api_key="YOUR_LANGTRACE_API_KEY")
lc_handler = lt.get_langchain_handler()  # example; check actual SDK name

prompt = PromptTemplate.from_template(
    "You are a helpful assistant. Answer the question: {question}"
)

llm = ChatOpenAI(model="gpt-4o", temperature=0)

chain = LLMChain(
    llm=llm,
    prompt=prompt,
    callbacks=[lc_handler],  # <- observability wired into this chain
)

Now, every time you call the chain, Langtrace will receive a trace of the full execution.

response = chain.invoke({"question": "Explain LangChain in one paragraph"})
print(response)

2. Tracing multi-step chains and tools

For more complex LangChain apps (e.g., Agents, Tools, RAG pipelines), attach the callback handler at the top-level entry point (agent, chain, or router chain). LangChain will propagate the callback to nested calls.

Example with tools:

from langchain.agents import initialize_agent, Tool
from langchain_community.tools import DuckDuckGoSearchRun
from langchain_openai import ChatOpenAI
from langtrace import Langtrace

lt = Langtrace(api_key="YOUR_LANGTRACE_API_KEY")
lc_handler = lt.get_langchain_handler()

search = DuckDuckGoSearchRun()
tools = [
    Tool(
        name="search",
        func=search.run,
        description="Useful for answering questions about current events.",
    )
]

llm = ChatOpenAI(model="gpt-4o", temperature=0)

agent = initialize_agent(
    tools=tools,
    llm=llm,
    agent="zero-shot-react-description",
    verbose=True,
    callbacks=[lc_handler],  # full agent trace goes to Langtrace
)

result = agent.invoke({"input": "What is Langtrace and why is it useful?"})
print(result)

In Langtrace, you’ll now see:

• A parent trace for the agent run
• Child spans for:
  • Each LLM call
  • Each tool invocation
  • Intermediate reasoning steps (where supported by LangChain integration)

---

Step 3: Add observability to a LlamaIndex app

LlamaIndex also supports structured tracing, and Langtrace integrates with it to capture each step in your index/query pipeline.

1. Wire Langtrace into LlamaIndex

After instantiating Langtrace, you typically register a callback manager or handler that works with LlamaIndex.

from llama_index import VectorStoreIndex, SimpleDirectoryReader
from llama_index.llms import OpenAI
from langtrace import Langtrace

lt = Langtrace(api_key="YOUR_LANGTRACE_API_KEY")
li_handler = lt.get_llamaindex_handler()  # example; use actual SDK name

documents = SimpleDirectoryReader("data/").load_data()
index = VectorStoreIndex.from_documents(
    documents,
    llm=OpenAI(model="gpt-4o"),
    callback_manager=li_handler,  # capture indexing traces
)

query_engine = index.as_query_engine(callback_manager=li_handler)
response = query_engine.query("What does Langtrace do?")
print(response)

Now each query produces:

• A trace for the query
• Steps for:
  • Document retrieval
  • Node scoring/reranking
  • LLM invocation
  • Final answer synthesis

2. Observing retrievers, rerankers, and tools

If you use advanced components (like custom retrievers, tools, or RAG pipelines), pass the Langtrace handler into their creation or call sites.

This makes it easy to answer questions like:

• Which documents were retrieved and why?
• How many tokens did each LLM call use?
• Which component is responsible for most latency?

---

Step 4: Viewing each step per request in Langtrace

Once your LangChain or LlamaIndex app is sending traces, you can explore every step inside the Langtrace UI.

Typical workflow:

1. Filter by route/user/trace
   • e.g., a specific endpoint like /chat or a specific user session.
2. Open the parent trace
   • See total latency, status, and metadata (e.g., user id, request id).
3. Inspect the timeline/tree
   • Each node is a step: LLM call, tool call, vector query, etc.
4. Click into a step to see:
   • Input prompt
   • Model and parameters
   • Tokens and cost (where supported)
   • Response text
   • Errors or warnings

This lets you debug issues like:

• “Why was this answer wrong?”
  → Check the retrieved documents and LLM prompt.

• “Why is this endpoint slow?”
  → Identify bottleneck steps (e.g., slow vector store or tool).

• “Is a particular user pattern causing failures?”
  → Filter traces by user ID and inspect their flows.

---

Step 5: Using observability to improve performance and safety

Observability is not just about debugging – it’s the foundation for systematic improvement.

With Langtrace traces in place, you can:

1. Run evaluations on real traces

Use recorded traces as test cases for:

• Accuracy and relevance
• Hallucination rates
• Safety and toxicity
• Policy compliance

You can plug these traces into evaluation pipelines and measure changes when you:

• Adjust prompts
• Swap models
• Update retrieval logic
• Add safety filters

2. Iterate on prompts and chain design

By looking at step-level data, you can refine:

• System prompts that lead to more grounded responses
• Retrieval parameters (e.g., top-k, similarity thresholds)
• Chain structure (e.g., splitting large tasks into smaller steps)

3. Improve security and robustness

Traces make it easier to:

• Detect prompt injection attempts or abuse patterns
• See when external tools are being called in unexpected ways
• Monitor for sensitive data leakage in prompts or outputs

This combination of observability + evaluations is the best way to iteratively improve both the performance and security of LangChain and LlamaIndex-based AI agents.

---

Step 6: Best practices for instrumenting LangChain/LlamaIndex apps

To get the most from Langtrace in a LangChain/LlamaIndex app, keep these practices in mind:

1. Instrument at the highest sensible level

   • Attach handlers to your top-level chains/agents/query engines so all nested calls are captured.

2. Propagate context/metadata

   • Include request IDs, user IDs, and business context as metadata so you can easily filter and debug traces.

3. Centralize initialization

   • Instantiate Langtrace once at startup and reuse the handlers across your app.

4. Monitor in staging before production

   • Use Langtrace in staging to catch design issues early, then roll into production with confidence.

5. Set up dashboards and alerts

   • Track latency, error rates, and key quality metrics derived from evaluations over your traces.

---

Framework & provider compatibility

Langtrace supports:

• Frameworks: LangChain, LlamaIndex, CrewAI, DSPy
• LLM providers: A wide range of major LLM APIs
• VectorDBs: Common vector databases used for retrieval and RAG

That means you can observe your entire RAG/agent system end-to-end, even if it spans multiple frameworks and infrastructure components.

---

Summary: adding observability step-by-step

To add observability to a LangChain/LlamaIndex app so you can see each step of the chain per request:

1. Create a Langtrace project and generate an API key.
2. Install the Langtrace SDK in your backend.
3. Instantiate Langtrace with your API key in your app’s initialization code.
4. Attach the Langtrace handlers:
   • To LangChain chains/agents via callbacks.
   • To LlamaIndex indices/query engines via callback managers.
5. Send real traffic and inspect traces in the Langtrace UI:
   • See parent traces and child steps for each request.
   • Inspect prompts, responses, tools, and latency.
6. Use those traces for evaluations and continuous improvement of performance and safety.

With this setup, every request through your LangChain or LlamaIndex app becomes fully observable, making it far easier to debug, optimize, and harden your AI agents in production.