Framework-agnostic agent tracing: Langfuse vs LangSmith vs other options

Most teams hit the same wall once their first agent demo is “working”: you can’t tell why a request took 30 seconds, why a tool wasn’t called, or why two similar questions produced different paths. That’s where agent tracing comes in—and where the choice between Langfuse, LangSmith, and open-standard options (like OpenTelemetry-based Arize Phoenix/AX) actually matters.

Quick Answer: Langfuse and LangSmith both give you strong LLM/agent traces, but they’re opinionated ecosystems anchored to their own SDKs and data models. If you want truly framework-agnostic agent tracing that survives framework changes and multi-agent architectures, favor OpenTelemetry + OpenInference–style tracing (e.g., Arize Phoenix in OSS, Arize AX in production) and use Langfuse/LangSmith where you’re comfortable with their SDK and runtime assumptions. The key is to standardize on open spans and traces so you can swap frameworks and providers without rewriting your observability layer.

Why This Matters

In early prototyping, tracing is a nice-to-have. In production, it’s the only way to answer basic reliability questions:

• Why did this agent call the wrong tool?
• Which model/prompt/router change caused today’s spike in hallucinations?
• Did this multi-agent workflow actually follow the expected path—or just stumble into a correct answer?

If your tracing is tied tightly to one framework’s abstractions or a proprietary SDK, you lose leverage the moment you:

• Switch from LangChain to custom orchestration
• Mix tools across OpenAI, Anthropic, and internal models
• Start composing multi-agent graphs that don’t fit a single framework’s mental model

A framework-agnostic approach anchors on open standards (OpenTelemetry spans, OpenInference naming) and treats your tracing backend as plumbing, not a custom integration every time you change your stack.

Key Benefits:

• Portability across frameworks: Instrument once using open standards, then swap orchestration frameworks (LangChain, LlamaIndex, custom agents) without redoing all your logging.
• End-to-end visibility for complex agents: Trace LLM calls, tool invocations, memory reads/writes, and multi-agent hops in a single graph so you can debug real production flows, not just single prompts.
• Evaluation-driven iteration: Attach offline/online evals and human annotations directly to traces, turning production edge cases into datasets and CI/CD gates—not just dashboards.

Core Concepts & Key Points

Concept	Definition	Why it's important
Framework-agnostic tracing	Capturing spans and traces in a way that doesn’t assume a specific agent/orchestration framework or vendor SDK.	Lets you keep one observability and evaluation layer as you evolve prompts, frameworks, and model providers.
Open standard tracing (OTEL + OpenInference)	Using OpenTelemetry for spans/traces plus conventions (like OpenInference) for LLM/agent metadata (prompts, responses, tool calls).	Avoids vendor lock-in, makes traces queryable in generic tooling, and lets you mix OSS (Phoenix) and SaaS (AX) without rewriting instrumentation.
Trace-linked evaluation	Attaching LLM-as-a-judge evals, code checks, and human labels directly to traces and spans.	Turns traces into regression tests and CI/CD signals so you can safely ship prompt/router/agent changes instead of eyeballing logs.

How It Works (Step-by-Step)

At a high level, here’s how I recommend thinking about “Langfuse vs LangSmith vs open-standard options” for framework-agnostic agent tracing.

1. Define your tracing substrate (OTEL or vendor SDK):

   • LangSmith: Primarily built around LangChain’s ecosystem; great if your agents are LangChain-first and you’re comfortable with their SDK and UI.
   • Langfuse: More general-purpose, with SDKs for multiple languages and frameworks; easier to adopt outside LangChain, but still its own data model and SDK.
   • OpenTelemetry + OpenInference (Arize Phoenix / AX): Use OTEL spans and OpenInference-like conventions to represent prompts, tool calls, context retrieval, and agents; works across frameworks and languages out of the box.

2. Map agent semantics to spans and traces:
   Regardless of tool, you need a consistent schema:

   • One trace per user request or session
   • Spans for:
     • Router decisions and policy checks
     • Each LLM call (with prompt + response payloads, redacted as needed)
     • Each tool call (with input/output)
     • Retrieval steps (vector search, DB query)
     • Multi-agent hops (agent A → agent B → tools) In Phoenix/AX, these become richly typed spans you can query, visualize as multi-agent graphs, and replay in a prompt playground.

3. Add evaluations and close the loop:
   Once tracing is in place, you layer evaluation:

   • Offline evals: LLM-as-a-judge (correctness, helpfulness, tool selection), code-based checks for deterministic logic, and structured scoring (e.g., JSON validity).
   • Online evals: Run evaluators on live traffic to catch hallucinations, policy violations, or tool-call mistakes in real time.
   • Human annotation queues: Route ambiguous or high-impact failures to humans; convert those labeled traces into golden datasets.
     Arize AX bundles all of this with CI/CD Experiments so you can gate releases; Phoenix gives you the open-source core for tracing + evals you can self-host.

---

To make this concrete, let’s break down the main options.

LangSmith: Best When You’re All-In on LangChain

One ecosystem. Deep LangChain integration for development, evaluation, and debug.

What it does well

• Tight LangChain integration:
  If your agents are LangChain-based, LangSmith basically “just works”—you get traces that align with LangChain constructs (chains, tools, retrievers) with little extra code.
• Friendly dev experience:
  Good for experimenting with chain/agent designs, visualizing step-by-step flows, and debugging early prototypes.
• Batteries-included evals for LangChain:
  You can run evals within that ecosystem, especially useful if you’re already using LangChain’s evaluation utilities.

Where it’s less framework-agnostic

• Framework coupling:
  The mental model and defaults assume LangChain. If you decide to:
  • Move to a custom agent loop,
  • Use a framework like Haystack, DSPy, or a homegrown orchestrator,
    you don’t get the same “native” experience. You’ll either wrap your logic in LangChain or invest in custom integration.
• Data model & SDK lock-in:
  You’re instrumenting to a LangSmith-specific API. Migrating away later means rewriting your logging layer.
• Limited open-standards story:
  You’re not building on OTEL spans or OpenInference-style conventions by default, which reduces portability to other observability tools.

Net-net: LangSmith is strong if:

• You’re committed to LangChain for the foreseeable future, and
• You want an integrated dev/debugging experience for that framework.

If you know your org will mix multiple frameworks or migrate over time, I’d be cautious about anchoring observability here.

Langfuse: Flexible, but Still a Proprietary Schema

Observability for LLM apps. More framework-neutral than LangSmith, good for simple traces and metrics.

What it does well

• Multi-framework SDKs:
  Works reasonably well with LangChain, LlamaIndex, and custom code. Easier to adopt as a “tracing + analytics” layer if you’re not LangChain-exclusive.
• LLM-native structure:
  It understands the basics: prompts, responses, latency, cost, and simple tree-structured traces.
• Quick time-to-value:
  If you just want “something” to visualize LLM calls and basic agent steps, you’ll get that quickly.

Where it falls short on framework-agnostic depth

• Custom data model:
  You’re instrumenting to Langfuse’s own schema and SDK. That’s more flexible than LangSmith but still a proprietary abstraction. If you later want to:
  • Integrate with existing OTEL pipelines,
  • Feed traces into enterprise observability stacks (Grafana, Datadog, etc.),
    you’ll be bridging two worlds.
• Limited multi-agent graph semantics:
  It handles traces and nested steps, but complex multi-agent graphs and cross-session state are harder to visualize and reason about compared to platforms that model sessions and agent graphs explicitly.
• Evaluation/CI/CD loop is thinner:
  You can log scores and metrics, but the full evaluation-driven CI/CD workflow (offline + online evals, annotation queues, experiment gating) is not as deep as a platform built specifically to “close the loop” between traces, evals, and releases.

Net-net: Langfuse is a solid middle ground when:

• You want more flexibility than LangSmith,
• You’re okay with a proprietary schema, and
• You don’t yet need OTEL-level integration or enterprise-grade eval/CI/CD.

Open Standard Tracing: Arize Phoenix & Arize AX

Built on open source & open standards. OTEL-based spans and traces with LLM/agent semantics via OpenInference-style conventions.

As someone who standardized OpenTelemetry tracing across our marketplace before we shipped our first agent to production, this is the path I’d choose again.

Arize Phoenix (open source, self-hosted)

• Open-source LLM tracing & evaluation:
  Self-hosted, no black-box frameworks. You get:
  • Traces and spans for LLM calls, tools, retrieval, and agents
  • Evaluations (LLM-as-a-judge, code checks) attached to traces
  • An interactive UI to inspect and debug traces
• OpenTelemetry-first:
  Phoenix speaks OTEL, so:
  • You can instrument in any language using standard OTEL SDKs
  • Traces can flow into your existing observability stack if needed
• OpenInference-style conventions:
  It uses consistent attributes and span naming for prompts, responses, and tools, rather than embedding everything in opaque payloads.

This is ideal if you want:

• Self-hosted control for regulated data,
• No data lock-in, and
• A clean path to production observability later (Arize AX).

Arize AX (AI & Agent Engineering Platform)

AX takes the Phoenix foundation and adds a production-grade environment:

• End-to-end tracing for agents:
  • Multi-agent graphs, session support, and long-running workflows
  • Spans for every step (LLM calls, tools, retrieval, memory) with prompt replay
  • 1 Trillion spans processed in real-world customer deployments
• Evaluation-driven CI/CD:
  • Offline evals (LLM-as-a-judge, code evals) to score traces and datasets
  • Online Evals to “Catch problems instantly with AI evaluating AI” on live traffic
  • CI/CD Experiments to gate prompt, model, and router changes so regressions are blocked before they hit users
• Prompt & agent iteration loop:
  • Prompt playground + prompt hub with trace replay
  • Annotation queues to turn edge cases into golden datasets
  • Experiments to compare prompts/models/parameters and pick the best, evaluated on real traces
• Enterprise-grade controls:
  • SOC 2 Type II, PCI DSS 4.0, HIPAA compliance
  • Data residency and multi-region self-hosting options
  • adb, a datastore designed for generative workloads (high-volume spans, evals, and datasets)

Most importantly for framework-agnostic tracing: AX is agnostic of vendor, framework, and language. You send OTEL spans and OpenInference-style attributes, not “LangChain objects” or “Langfuse runs.”

Net-net: If you want a tracing + evaluation layer that survives framework churn, multi-agent complexity, and enterprise governance, this is the cleanest path.

Other Options & Patterns (And When They Break Down)

You’ll also see teams try to roll their own with:

• Custom logging + dashboards:
  Log JSON to Elasticsearch, BigQuery, or a data lake; build Grafana dashboards. This works for:

  • Early cost/latency monitoring
  • High-level trend analysis
    It breaks down when you need:
  • Span-level semantics (parent-child relationships, concurrency)
  • Interactive trace inspection and replay
  • Tight coupling between traces and evals

• Generic APM tools (Datadog, New Relic, etc.):
  Good for infra and API-level metrics, but not built for:

  • Prompt/response inspection
  • Tool-call semantics
  • LLM-specific evals and annotation workflows
    You can pipe OTEL spans through them, but you’ll still want an LLM-aware platform for daily agent work.

Pattern I recommend:
Use OTEL as your base, send LLM/agent spans to a generative-aware platform (Phoenix/AX), and optionally mirror aggregated metrics to your traditional APM.

Common Mistakes to Avoid

• Mistake 1: Binding observability to a single framework’s abstractions
  Teams wire everything into LangChain + LangSmith, then realize they can’t easily:

  • Swap to custom orchestration,
  • Run agents in a different runtime, or
  • Mix in non-LangChain tools.
    How to avoid it: Instrument at the span level with OTEL. Treat frameworks as implementation details. Even if you use LangSmith or Langfuse, keep an OTEL-based pathway ready so you’re not rewriting everything later.

• Mistake 2: Treating tracing and evaluation as separate systems
  Logging traces in one tool and evaluation results in another makes it impossible to ask, “What changed in the agent’s path on failed evals?”
  How to avoid it: Choose a platform where:

  • Evals are attached directly to traces and spans
  • You can filter traces by eval score
  • CI/CD Experiments run on real trace data (like in Arize AX)

Real-World Example

At our marketplace, we started with a LangChain-based routing agent and basic logging. Every incident review sounded the same: “We can’t see what happened in the middle.”

We moved to an OTEL + OpenInference-style model with Arize Phoenix first:

• Defined a schema: one trace per user request, spans for each LLM call, tool invocation, and retrieval step.
• Instrumented agents in Python with standard OTEL SDKs and OpenInference-style attributes (prompt, response, model, tool_name, etc.).
• Ran Phoenix to visualize traces and attach LLM-as-a-judge evals for:
  • Answer correctness
  • Tool selection appropriateness
  • Policy adherence

Later, as traffic and regulatory pressure increased, we upgraded to Arize AX:

• Mirrored Phoenix-style spans into AX via OTEL exporters.
• Built CI/CD Experiments: any prompt or router change has to beat the current baseline on our eval datasets, or it won’t ship.
• Set up Online Evals + alerts on hallucination and tool-call correctness so we can “catch problems instantly” on live traffic.

We still use other frameworks (some teams experiment with LangSmith; another team uses Langfuse for a narrow use case), but our core reliability and SLO posture rests on OTEL-based tracing and evaluation in AX. When one team rewrote an agent with a different framework, we didn’t touch the dashboards or CI/CD pipelines—just the instrumentation glue.

Pro Tip: Standardize a tracing schema before tooling. Decide what a “request trace” and “tool span” mean in your org, codify it via OTEL and OpenInference-like attributes, and only then layer Langfuse/LangSmith/Phoenix/AX on top. Tools will change; your trace schema shouldn’t.

Summary

Framework-agnostic agent tracing isn’t about picking a winner between Langfuse and LangSmith. It’s about designing an observability and evaluation layer that:

• Survives framework and provider changes,

• Makes every agent step (LLM call, tool, retrieval, memory, multi-agent hop) inspectable, and

• Feeds directly into evaluation-driven CI/CD so you can “Ship Agents that Work” instead of agents that merely demo.

• LangSmith is great if you’re deeply committed to LangChain and want tight integration in that ecosystem.

• Langfuse is a flexible proprietary option for LLM/agent observability when you don’t need deep OTEL integration.

• OpenTelemetry + OpenInference with Arize Phoenix/AX gives you open-standard, framework-agnostic tracing and a full loop of tracing → evaluation → CI/CD → monitoring, backed by production scale and compliance.

If you want to avoid re-instrumenting every time you change frameworks, anchor on OTEL spans and a platform that treats traces, evals, and experiments as one system.

Next Step

Get Started