LLM observability + evals: cloud SaaS vs self-hosted/hybrid in our VPC (tradeoffs, security, effort)

Most teams don’t start by asking “cloud vs self-hosted?” They start by hitting a wall: agents that work in staging silently fail in production, and nobody can replay what happened. That’s where LLM observability and evals come in—and where the cloud SaaS vs self-hosted/hybrid decision actually matters: it determines how fast you get to that replayable reality, and how tightly you can lock it down inside your VPC.

Quick Answer: LangSmith gives you both options: a fully managed cloud SaaS for fastest time-to-value, and self-hosted/hybrid deployments that keep traces, prompts, and eval data in your own VPC while still using the same trace-first workflows. The right choice depends on your data sensitivity, governance requirements, and how much operational effort you’re willing to own.

The Quick Overview

• What It Is: LangSmith is a trace-first LLM observability and evaluation platform to monitor, debug, and improve agents and complex LLM workflows, with deployment options ranging from cloud SaaS to self-hosted in your VPC.
• Who It Is For: Engineering, data, and platform teams shipping production agents—RAG systems, copilots, workflow and tooling-heavy agents—who need replayable traces, systematic evals, and reliable deployments with enterprise controls.
• Core Problem Solved: Non-deterministic agent behavior, long context windows, and branching tool calls make it impossible to pre-plan every outcome. LangSmith captures what your agents actually did in production, turns those traces into datasets and evals, and gives you a governed runtime to deploy agents that don’t silently regress.

How It Works

LangSmith sits alongside your agent stack, instrumenting every run as a structured trace. Those traces become the backbone for debugging, analytics, and evaluation. From there, you can iterate on prompts, tools, and policies—and then ship agents on a durable runtime with versioning, memory, and rollbacks. The same primitives are available whether you run LangSmith as cloud SaaS or inside your VPC.

1. Instrument & Trace (Build / Observe):

   • Use the LangSmith SDKs (Python, TypeScript, Go, Java) or OpenTelemetry to send traces for any agent framework—LangChain, LangGraph, Deep Agents, or your custom stack.
   • Capture every step: LLM calls, tools, RAG retrieval, control flow. See timelines and message threads that show exactly what happened and in what order.

2. Turn Traces into Datasets & Evals (Evaluate):

   • Convert production traces into datasets with one click or via API.
   • Define evaluators (automated metrics, LLM-as-judge, human-in-the-loop via annotation queues).
   • Use Align Evals to calibrate LLM judges with human corrections and few-shot examples so scores reflect your domain and policy, not generic “quality.”

3. Deploy & Govern (Deploy):

   • Promote agents into LangSmith Deployment (or integrate with your own runtime) to get durable checkpointing, memory, conversational threads, and exactly-once execution.
   • Manage versions and rollbacks, route traffic, and keep humans in the loop via approval flows for sensitive tools.
   • Choose deployment: cloud SaaS, hybrid, or self-hosted in your VPC—same APIs, different control plane.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Trace-First Observability	Captures structured runs with timelines, tool calls, messages, and intermediate state for every agent invocation.	Makes debugging and root-cause analysis possible for long, branching agent workflows—code alone isn’t enough.
Evaluations & Align Evals	Runs offline and online evals (including multi-turn and LLM-as-judge) on datasets built from real traces, calibrated with human feedback.	Lets you measure quality, catch regressions before production, and continuously improve agents using the same criteria as your SMEs.
Durable Deployment Runtime	Provides stateful, long-running agent execution with memory, checkpointing, exactly-once semantics, and versioned deployments.	Lets you safely run serious agents in production, with rollbacks and admin controls instead of one-off scripts and ad hoc infra.

Cloud SaaS vs Self-Hosted/Hybrid: How It Actually Plays Out

You’re balancing three things:

• Security & data residency: Where do traces, prompts, and evaluation data live?
• Operational effort: Who maintains the observability stack and runtime?
• Iteration speed: How quickly can you instrument, debug, and evolve agents?

LangSmith is built to let you slide along that spectrum without changing your development model.

Cloud SaaS: Fastest Way to See What Your Agents Are Doing

Cloud LangSmith is the default for teams that want to move quickly while still having enterprise-grade controls.

Mechanics:

• Hosted by LangChain with US/EU data residency options.
• You instrument your agents via SDKs/OpenTelemetry; traces, datasets, and evals are stored in LangSmith’s managed environment.
• You can still keep model and tool data in your own VPC; only metadata and payloads you choose to log flow to LangSmith.

What you get:

• Minimal setup, no infra:

  • No cluster provisioning, backups, patching, or scaling to handle “we just hit 1B events/day” moments.
  • Immediate access to new features—Polly (the built-in assistant), Insights Agent, new evaluators—without upgrades.

• Enterprise controls baked in:

  • SSO/SAML, SCIM, audit logs, RBAC/ABAC for granular access control.
  • Clear data posture: LangSmith does not use your data to train models.
  • Regional hosting (US/EU) to align with data residency needs.

• Operational performance:

  • The same stack that handles over 1 billion events per day for 6K+ customers, including 5 of the Fortune 10.
  • You benefit from the tuning we’ve already done under real production loads.

Tradeoffs:

• Some sensitive teams (e.g., regulated industries, strict data-sovereignty rules) may not be allowed to send full prompts, outputs, or user data outside their VPC—even with redaction and controls.
• Network and compliance reviews are simpler than self-hosting, but they still exist; legal and security teams will want to vet the SaaS.

Self-Hosted / Hybrid in Your VPC: Maximum Control, More Responsibility

For teams with strict data boundaries or a “nothing leaves the VPC” mandate, LangSmith can be deployed in a self-hosted or hybrid model.

Mechanics:

• You deploy LangSmith inside your own cloud environment (AWS, GCP, Azure) or on-prem.
• Traces, datasets, eval results, and deployment metadata stay within your VPC.
• You can still integrate with external models or gateways (OpenAI, Anthropic, custom MCP servers) using your existing network controls.

What you get:

• Full data residency and isolation:

  • No customer data leaves your environment; easier alignment with internal policies and regulators.
  • You can co-locate LangSmith with your existing data stores and key management systems.

• Custom governance stack:

  • Integrate with your enterprise SSO, IAM, VPC peering, and private networking patterns exactly the way you want.
  • Layer additional controls (e.g., DLP, internal audit pipelines) on top of traces and eval datasets.

• Same trace-first workflows:

  • You still get run timelines, threads, annotation queues, Align Evals, versioning, and rollbacks.
  • Framework agnostic: instrument any agent stack using the same SDKs.

Tradeoffs:

• You own reliability and scaling:

  • Capacity planning for trace volume (ingesting millions/billions of events).
  • HA, backups, disaster recovery, observability for the observability system.
  • Upgrade and migration cycles when new features ship.

• Higher initial lift:

  • Terraform, Helm, or equivalent infra-as-code.
  • Security hardening, internal compliance sign-offs, and ongoing maintenance.

Hybrid Patterns

A lot of large customers land in the middle:

• Hybrid data split:

  • Sensitive fields (PII, financial data) are masked or not logged; only structural trace data and anonymized content leave the VPC.
  • Or: core traces stay self-hosted, but you use cloud LangSmith for non-sensitive projects or sandboxes.

• BYO models and tools, regardless of deployment:

  • Whether cloud or self-hosted, you bring your own models, vector stores, and tools; LangSmith doesn’t force a specific LLM provider.

Features & Benefits in the Cloud vs Self-Hosted Context

Core Feature	What It Does	Primary Benefit
Framework-Agnostic Tracing	Integrates via SDKs and OpenTelemetry with any agent framework.	Lets you standardize observability across multiple teams and stacks without lock-in.
Annotation Queues & Align Evals	Routes traces to SMEs for review, then uses corrections to calibrate LLM-as-judge evals.	Turns real production behavior into reliable, domain-specific quality scores, not generic benchmarks.
Deployment & Runtime Controls	Provides a versioned agent registry, rollbacks, memory, threads, and tool-level approvals.	Lets you safely give agents more autonomy while keeping humans and admins in control—especially important in regulated environments.

Ideal Use Cases

• Best for “move fast, prove value” (cloud SaaS): Because you can instrument in a day, start capturing traces immediately, and get your first eval runs and regression tests running without touching infra. Ideal if you’re still validating the business impact of agents or you’re early in rollout.
• Best for “regulated, high-sensitivity workloads” (self-hosted/hybrid): Because you keep all observability and eval data in your own VPC, integrate tightly with internal security tooling, and satisfy strict policies around data residency and tenancy while still getting full trace and eval capabilities.

Limitations & Considerations

• Cloud SaaS data boundaries:
  Even with regional hosting and a “no training on customer data” posture, some organizations simply cannot send any user-level or domain-sensitive content to a third-party SaaS. In those cases, use field-level redaction, payload minimization, or move to a self-hosted deployment.

• Self-hosted operational overhead:
  Running and scaling an LLM observability and eval stack isn’t free. You’ll need an internal owner for upgrades, capacity, and SRE-style reliability. If your platform team is already overloaded, start with cloud, learn what “good” looks like, then decide whether to bring it in-house.

Pricing & Plans

LangSmith is designed to be accessible for small teams while scaling to Fortune 500 requirements.

• Cloud SaaS:

  • Seat-based pricing for users plus pay-as-you-go usage based on traces/events.
  • Base plans with shorter trace retention (e.g., 14 days) and higher tiers with extended retention (up to ~400 days) and more advanced admin controls.
  • Good fit if you want to “pay for what you use” and avoid infra.

• Enterprise / Self-Hosted or Hybrid:

  • Custom contracts tailored to your deployment model (VPC, hybrid, regional), volume, and security posture.
  • Includes options like dedicated environments, custom retention, and deeper integration support.
  • Best if you’re standardizing LangSmith as the agent observability layer across many teams.

(For specific numbers and deployment options, talk to sales; pricing is tuned around your volume and compliance needs.)

• Growth/Team Plan (Cloud): Best for product and engineering teams needing full observability and eval workflows without owning infra, plus standard enterprise features (SSO, SOC2, etc.).
• Enterprise/VPC Plan: Best for platform orgs and regulated enterprises needing strict data residency, self-hosting or hybrid models, extended retention, and deep governance controls.

Frequently Asked Questions

Can we start on cloud LangSmith and later move to self-hosted in our VPC?

Short Answer: Yes, you can start in the cloud and migrate to self-hosted or hybrid when you’re ready.

Details:
Most teams don’t know their final deployment posture on day one. That’s why LangSmith is designed so the same primitives—traces, datasets, evals, deployments—exist across cloud and self-hosted. You can:

• Start with cloud to instrument your agents, learn what you need to log, and establish your eval strategy.
• Use that learning to design a self-hosted architecture with correct sizing and data boundaries.
• Migrate projects and datasets, keeping your test suites and eval definitions intact so you can validate behavior after the move.

This avoids the common anti-pattern of building a bespoke internal observability system only to discover it can’t handle multi-agent traces, alignment evals, or long-running workflows.

How do you protect our data in cloud SaaS—and do you ever train on it?

Short Answer: LangSmith does not use your data to train models, and it’s built with enterprise security controls and regional hosting.

Details:
From day one we assumed agents would touch sensitive data. In the cloud deployment:

• Data use:

  • LangSmith does not use your prompts, outputs, traces, or evals to train any models.
  • Data is processed only to provide observability, evals, analytics, and runtime functions.

• Security & governance:

  • SSO/SAML, SCIM, RBAC/ABAC, and audit logs to control and monitor access.
  • Encryption in transit and at rest; options for US/EU residency.
  • Support for redaction, selective logging, and minimizing what you send.

If those controls aren’t enough for your internal policy, the self-hosted or hybrid option lets you keep everything inside your own security perimeter while using the same feature set.

Summary

You can’t fix what you can’t replay, and you can’t trust agents in production without measuring their behavior over time. LangSmith gives you a trace-first observability and evals platform that works with any agent stack—and you choose where it runs.

• Cloud SaaS is the fastest on-ramp: minimal setup, proven at scale (1B+ events/day, 6K+ customers), and full-featured tracing, evals, and deployment with strong enterprise controls.
• Self-hosted/hybrid in your VPC gives you maximum control and data residency, at the cost of owning reliability, scaling, and upgrades.

Either way, you get the same core workflow: trace → dataset → evals (with Align Evals) → deploy → monitor → iterate, backed by a runtime that’s built for long-running, stateful agents with human-in-the-loop approvals and rollbacks.

Next Step

Get Started