How do we set up online evals in LangChain LangSmith on real user conversations (and alert on quality drops)?

Most teams don’t discover quality drops in their AI agents until users complain. Online evals in LangSmith are how you turn real user conversations into a continuous “smoke detector” for quality drift—scoring production traces in real time, surfacing failures, and triggering alerts when things regress.

Quick Answer: You set up online evals in LangSmith by (1) tracing real user runs into a project, (2) configuring evals in the UI with filters and sampling on those traces, (3) attaching LLM, heuristic, or human evaluators, and (4) wiring dashboards and alerts to notify you when scores drop or failure patterns emerge.

---

The Quick Overview

• What It Is: Online evals in LangSmith are real-time evaluations that run on your live production traces—scoring agent outputs and behavior against criteria you define (accuracy, safety, tone, etc.).
• Who It Is For: Teams running LangChain, LangGraph, Deep Agents, or any agent stack in production who need to catch regressions and quality drift before they turn into tickets.
• Core Problem Solved: LLM agents fail silently. Logs don’t show whether the answer was actually good. Online evals add continuous, automated judgment on every (or sampled) real user conversation so you can detect and debug quality issues quickly.

---

How It Works

At a high level, you:

1. Send traces from real user conversations to LangSmith.
   Instrument your agent so every run, tool call, and message thread is captured as a trace.

2. Configure online evals in the UI.
   In your LangSmith project, you create evals, choose which runs to target (via filters), set sampling, and pick evaluator types (LLM-as-judge, heuristics, or human).

3. Monitor and alert on quality signals.
   LangSmith computes eval scores on incoming production traces. You track them in dashboards, use the Insights Agent to spot patterns, and attach alerts to detect quality drops in real time.

1. Instrument your agent to send real user traces

Online evals run on traces. If you don’t trace it, you can’t evaluate it.

• Choose your integration:
  • LangChain, LangGraph, Deep Agents: enable native LangSmith tracing.
  • Any other stack: use LangSmith SDKs for Python, TypeScript, Go, or Java, or OpenTelemetry.
• Attach runs to a project:
  • Configure your environment with a LANGSMITH_PROJECT (or equivalent) so all production conversations land in a specific project, e.g., prod-assistant.
• Capture full context:
  • User input and model outputs.
  • Tool calls and results.
  • System / developer messages and policies.
  • Metadata you may want to filter on later (tenant IDs, model versions, routes, experiment flags, etc.).

Once traces are flowing, you have a live stream of real user conversations that online evals can target.

2. Configure online evals in the LangSmith UI

Online evals are configured at the project level.

1. Navigate to your tracing project.
   Open LangSmith → select your production project (e.g., prod-assistant).

2. Create a new evaluation.

   • Choose Online Eval (not just offline/dataset eval).
   • Give it a clear name like prod-answer-correctness-v1.

3. Set filters to target the right runs. You rarely need to evaluate everything. Use filters so you’re scoring the runs that matter:

   • Only successful user-facing runs (exclude tool-only runs, health checks).
   • Only specific routes or chains (e.g., order_status_agent).
   • Only certain tenants / customers (for staged rollouts).
   • Only specific models (to compare model variants).

4. Configure sampling.

   • Sampling rate: e.g., 5–10% of eligible runs is usually enough to spot drift without exploding eval cost.
   • Stratified sampling: ensure coverage across key segments (language, product line, route) by encoding them as metadata and filtering / splitting evals accordingly.

5. Choose evaluation types. LangSmith supports multiple evaluator types:

   • LLM-as-judge evaluators

     • Use an LLM to score a conversation or answer against criteria:
       • Correctness / groundedness
       • Helpfulness / completeness
       • Safety / policy compliance
       • Tone / style
     • You define:
       • The prompt / rubric (what “good” looks like).
       • Output format (e.g., 1–5 score + explanation).
     • For more reliability, calibrate these with Align Evals using human corrections and few-shot examples.

   • Heuristic evaluators

     • Code-based checks such as:
       • Did the answer include a reference/citation?
       • Is JSON valid? Did code compile?
       • Did the agent call the required tools?
       • Response length / latency constraints.
     • These are great for cheap, deterministic guardrails.

   • Human evaluation via annotation queues

     • Route a subset of traces into Annotation Queues for subject matter experts.
     • Use consistent rating forms and rubrics.
     • Human scores can then be used to:
       • Directly monitor high-risk flows.
       • Train and calibrate LLM-as-judge evaluators.

You can combine these—e.g., run cheap heuristics on 10% of traffic, LLM-as-judge on 5%, and human review on 0.1% of high-risk traces.

3. Attach evaluators and run them online

Once you’ve defined evaluator types and filters, online evals run automatically:

• As new runs arrive in your project and match filters/sampling, LangSmith:
  • Fetches the trace.
  • Feeds the relevant context to the evaluator (LLM prompt, heuristic function, or human queue).
  • Stores evaluator results back on the run (scores, labels, explanations).

This turns your production project into a live dataset of:

• Traces (what actually happened).
• Scores (how well it behaved).
• Metadata (model version, route, user segment, etc.).

---

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Online Evals on Production Traces	Continuously evaluates real user conversations as they happen	Detects drift and regressions in real time instead of after a wave of support tickets
Multi-Type Evaluators (LLM, Heuristic, Human)	Supports LLM-as-judge, code-based checks, and human annotation queues	Lets you mix cheap automated checks with high-signal human feedback for robust quality signals
Trace-First Insights & Alerts	Stores eval scores on traces, powers dashboards, Insights Agent, and alerts	Gives you end-to-end visibility, root-cause analysis, and automated notifications on quality drops

---

Ideal Use Cases

• Best for production AI assistants / agents:
  Because it continuously evaluates live user conversations for correctness, safety, and tone, and surfaces problematic runs with full trace context for debugging.

• Best for teams iterating on models, prompts, or routing:
  Because it gives you side-by-side quality metrics on different versions in real time, letting you catch regressions during experiments instead of after rollout.

---

Limitations & Considerations

• LLM-as-judge is still probabilistic:
  LLM evaluators can be noisy or biased. Use Align Evals to calibrate them against human-annotated data, measure agreement, and update your prompts/criteria. Always keep some human review on critical flows.

• Evaluation cost and latency:
  Running evals on every production run can be expensive and slow. Use sampling, heuristic filters, and targeted online evals so you’re spending evaluation budget where it matters most (high-value users, risky actions, new releases).

---

Pricing & Plans

LangSmith pricing is designed so you can start small and scale online evals as your agent matures.

• Team / Self-Serve Plans:
  Best for product and engineering teams needing:

  • Seat-based access to tracing, evals, and basic dashboards.
  • Pay-as-you-go usage for traces and eval calls.
  • Standard data retention (e.g., shorter default windows) sufficient for most pilots and early production.

• Enterprise Plans:
  Best for larger organizations needing:

  • Higher-volume evals across multiple agents and business units.
  • Extended retention (400-day or custom) for compliance, long-term benchmarking, and alignment work.
  • Deployment flexibility (US/EU residency, hybrid, or self-hosted), plus SSO/SAML, SCIM, RBAC/ABAC, audit logs, and strict data isolation.
    LangSmith does not use your data to train models.

(For exact pricing, volume tiers, and retention details, talk to sales.)

---

Setting up alerts on quality drops

Online evals give you scores; alerts turn those scores into a safety net.

Step 1: Build quality dashboards

In LangSmith:

• Create dashboards that track:
  • Average eval score over time (e.g., 1–5 correctness).
  • Distribution of scores (how many low scores per day).
  • Breakdown by:
    • Model version.
    • Route / chain / tool.
    • Tenant or customer segment.
• Add operational metrics alongside:
  • Latency, error rate, and cost.
  • Volume of runs per route.
  • Percentage of runs being evaluated (sampling coverage).

This gives you a living quality control panel.

Step 2: Configure alerts on eval metrics

Use LangSmith’s monitoring capabilities to attach alerts to key metrics:

• Threshold-based alerts:
  • Notify when average eval score drops below a threshold for a window, e.g.:
    • “Alert if 1-hour rolling average correctness < 3.8.”
    • “Alert if more than 5% of runs in the last 30 min scored ≤ 2.”
• Trend / change alerts:
  • Alert on deltas relative to baseline:
    • “Alert if correctness drops by >0.5 vs last week’s average.”
    • “Alert if safety violations increase by 2x day-over-day.”

When an alert fires, link back into the filtered set of runs that triggered it so on-call teams can immediately inspect traces.

Step 3: Use Insights Agent for pattern extraction

When online evals flag problematic traces at scale, raw logs are overwhelming. LangSmith’s Insights Agent helps by:

• Digesting large volumes of traced runs and eval scores.
• Surfacing patterns:
  • Common failure intents or user segments.
  • Specific tools or routes where scores are low.
  • Prompts or models correlated with regressions.
• Generating an “executive summary” of:
  • What’s failing.
  • How often.
  • Where to focus fixes.

Because this is built on top of trace data, you can always click from the summary down into exact runs for verification.

---

Example: End-to-end online eval setup on real conversations

To make this concrete, here’s a typical workflow for a production support assistant:

1. Instrument tracing

   • All user chats and tool calls are traced into project prod-support-assistant.

2. Define key evals

   • support-answer-correctness-v1 (LLM-as-judge, 1–5 score).
   • support-policy-compliance-v1 (LLM-as-judge for safety + heuristic to detect forbidden words).
   • support-escalation-needed-v1 (LLM-as-judge: should this have been escalated?).

3. Targeted online eval config

   • Filters:
     • type = 'user-facing'
     • route in ['faq_agent', 'order_agent']
   • Sampling:
     • 10% for correctness.
     • 10% for compliance.
     • 5% for escalation detection.
   • Evaluators:
     • LLM-as-judge prompts calibrated with 200 human-labeled examples via Align Evals.

4. Dashboards

   • One board showing:
     • Correctness and compliance over time.
     • Escalation-needed vs actual escalations.
     • Metrics split by model version.

5. Alerts

   • Pager/Slack alert when:
     • Correctness < 4.0 for 30 minutes.
     • Policy-compliance low-score runs > 1% in the last hour.
     • Escalation-needed score > actual escalation rate by 2x (agent is missing handoffs).

6. Operational loop

   • When an alert hits, on-call engineer:
     • Opens the dashboard → filters to the failing segment.
     • Drills into traces → reviews the lowest-scoring runs.
     • Uses Annotation Queues to get SME labels on borderline cases.
     • Updates prompts, tools, or routing.
     • Runs offline evals on regression datasets before deploying changes.
     • Watches online eval metrics to confirm recovery.

This is the “observe → evaluate → fix → verify” loop we see work in practice.

---

Frequently Asked Questions

How do online evals in LangSmith differ from offline evals?

Short Answer: Online evals run on live production traces in real time; offline evals run on fixed datasets for controlled regression testing.

Details:
Offline evals are great before you ship: you build a dataset, run evaluators, and compare versions in a controlled environment. Online evals, by contrast, continuously sample and score real user conversations as they happen. They catch drift from new user behavior, changing content, or external systems. In practice, teams use both: offline evals for pre-release regression checks, online evals as the production “smoke detector.”

---

Do I need to use the LangChain framework to use online evals?

Short Answer: No. LangSmith is framework agnostic and works with any agent stack.

Details:
While LangSmith has native tracing for LangChain, LangGraph, and Deep Agents, online evals operate on traces, not framework-specific code. You can instrument any stack using LangSmith SDKs (Python, TypeScript, Go, Java) or OpenTelemetry, send runs into a project, and configure online evals on those traces exactly the same way. This is intentional: we want you to bring your own models, tools, and frameworks, and still get the same trace-first observability and evaluation loop.

---

Summary

Online evals in LangSmith are how you turn real user conversations into a continuous quality signal. By tracing production runs, attaching LLM, heuristic, and human evaluators, and monitoring scores in dashboards and alerts, you move from reactive debugging to proactive quality management. You see exactly what your agent did, how well it performed, and where it’s drifting—then use those traces as fuel to improve prompts, tools, and models without shipping regressions.

---

Next Step

Get Started