How do teams catch hallucinations and unsafe outputs in production without manually reading thousands of conversations?

Most teams discover the hard way that you cannot QA your way out of hallucinations and unsafe outputs by manually reading transcripts. At any meaningful scale, manual review becomes noise—you need automated eyes on every span, every response, and every conversation, with humans stepping in only where judgment really matters.

Quick Answer: Teams catch hallucinations and unsafe outputs in production by turning every LLM/agent response into a scored event—using online evaluations (“LLM as a Judge”), real-time guardrails, and targeted human annotation queues instead of manual transcript review. With OTEL-based tracing, evaluators, and monitors, you can scan 100% of traffic, trigger alerts on risky behavior, and send only the most ambiguous or impactful cases to humans for review.

Why This Matters

If you’re shipping agents into customer-facing or regulated workflows, “we’ll spot bad outputs if they happen” is not a safety strategy—it’s a hope strategy. Hallucinations, PII leaks, and toxic content don’t show up in neat batches; they hide in long-tail slices and rare edge cases. Without automated detection and evaluation, you either:

• Overreact and lock down the system so tightly it stops being useful, or
• Under-react and end up with brand, compliance, or safety incidents that were visible in the data but never surfaced.

A production-ready approach uses tracing, evaluations, and guardrails to continuously watch for unsafe behavior, quantify risk, and feed real incidents back into your training and testing loops.

Key Benefits:

• Scale beyond manual review: Automatically evaluate millions of responses for hallucinations, toxicity, PII, and policy violations without hiring an army of reviewers.
• Catch problems in real time: Use online evaluations and guardrails to trigger alerts or block responses the moment metrics drift out of bounds.
• Continuously improve quality: Turn detected failures into datasets, experiments, and updated prompts/agents so the system gets safer and more accurate over time.

Core Concepts & Key Points

Concept	Definition	Why it's important
Online Evals (LLM as a Judge)	Automated scoring of each response in production using LLM-based and code-based evaluators (e.g., hallucination risk, PII presence, toxicity).	Lets you monitor every interaction continuously, catch regressions instantly, and quantify risk without reading every conversation.
Real-Time Guardrails	Input/output checks and policies that run before a response is shown, blocking, rewriting, or escalating unsafe content.	Provides a safety net for high-risk cases (PII, hate, self-harm, compliance violations) where “log and review later” isn’t acceptable.
Annotation Queues & Datasets	Workflows where suspicious or ambiguous responses are routed to humans, labeled, and saved as curated datasets.	Converts production edge cases into golden test sets for future evals, experiments, and regression tests so your system keeps getting safer.

How It Works (Step-by-Step)

One platform. Production traces become signals you can evaluate, monitor, and act on.

At a high level, the loop looks like this:

1. Trace every request and response with OTEL
2. Evaluate outputs automatically for safety and correctness
3. Monitor and intervene in real time with guardrails and alerts
4. Feed incidents back into experiments and prompt/agent updates

Here’s the breakdown of how teams implement this in practice.

1. Instrument: Trace every span and session

You can’t catch what you can’t see.

• Use OpenTelemetry-based tracing to log every request, tool call, and model response as spans and traces.
• Adopt OpenInference conventions so prompts, model metadata, tools, and outputs are structured and consistent—no proprietary tracing format, no lock-in.
• Capture:
  • User input, context (RAG documents, retrieved facts)
  • Model prompts and responses (including intermediate chain-of-thought where allowed)
  • Tools called, parameters passed, and tool outputs
  • System decisions (routing, fallback models, escalation paths)

In Arize AX and Arize Phoenix, these traces become a navigable graph of the entire agent path. You can replay and debug exactly how the agent arrived at a problematic answer.

Why this matters: Hallucinations and unsafe outputs are often tied to subtle path issues: wrong tool selection, missing retrieval, or a brittle prompt. Tracing lets you pinpoint root causes instead of just seeing the last message.

2. Evaluate: Use “LLM as a Judge” + code checks on every response

Once you have full traces, you layer on evaluations.

LLM-based evaluators (online + offline):

For each LLM output, you can run structured evaluators that answer questions like:

• “Does this response rely on facts not present in the retrieved context?” → Hallucination risk
• “Does this contain PII (email, phone, SSN, address)?” → PII detection
• “Does this contain toxic, hateful, or self-harm content?” → Toxicity / safety
• “Did the agent follow the instructions and respect the system policy?” → Policy adherence

Using Arize, these show up as evaluator scores and labels attached to spans:

• Binary labels: safe / unsafe, hallucinated / grounded
• Scaled scores: 0–1 or 1–5 for severity, helpfulness, faithfulness
• Rubrics: multi-criterion scoring (e.g., safety + helpfulness + relevance)

You can start with open models and open-source eval templates—no black-box eval systems required.

Code-based evals for deterministic checks:

For many unsafe cases, you don’t need an LLM judge at all; simple deterministic checks are more reliable and cheaper:

• Regex and pattern checks for PII (emails, credit cards, phone numbers)
• Keyword and phrase lists for restricted topics (compliance, health, finance)
• JSON schema validation for responses that must be structured
• “Must not mention X” guardrails for system policies

In Arize, these show up as code evals tied to the same spans as LLM evals, so you can blend hard rules with fuzzy judgments.

Why this matters: Evaluations turn raw text into metrics you can monitor, alert on, and use in CI/CD. Without evals, you’re stuck sampling conversations and hoping you spot the bad ones.

3. Guard: Block or reshape unsafe outputs in real time

Detection is only half the story. For certain risk categories, you need real-time intervention.

Teams typically layer real-time guardrails on top of evaluations:

• Input validation:

  • Detect prompt injection, jailbreak attempts, or unsupported languages before the model sees them.
  • Strip or rewrite inputs that try to override system instructions.

• Output validation:

  • Run fast code checks (PII regex, keyword filters) synchronously.
  • Optionally run a lightweight LLM-based safety classifier inline.

• Response strategies on violation:

  • Block and show a safe fallback message.
  • Rewrite: ask the model to respond safely (e.g., “explain we can’t advise on X but suggest Y”).
  • Escalate: route to a human or specialized workflow.

Arize’s Online Evals and Monitor capabilities let you define thresholds and policies like:

• “If hallucination score > 0.8 or safety label = unsafe → trigger alert + log for review”
• “If PII detected → block response + create annotation task”

Why this matters: For sensitive domains (health, finance, legal, minors), “we’ll catch unsafe outputs later in a dashboard” isn’t enough. Guardrails enforce hard stops where your risk tolerance is zero.

4. Monitor: Dashboards, drift, and alerts on safety metrics

With evaluations running on traces, you can treat safety like any other production SLO.

In Arize AX, teams typically set up:

• Dashboards tracking:

  • % responses with hallucination risk above threshold
  • Safety / toxicity rate
  • PII detection rate
  • Policy violation rate by route/model/prompt version
  • Latency and token/cost metrics alongside safety metrics

• Custom metrics and slices:

  • Safety by user segment, language, country, or product area
  • Hallucination rate by retrieval source or document collection
  • Toxicity by channel (chat vs. email vs. voice-transcription)

• Alerts and SLOs:

  • “Alert if hallucination_rate > 2% over 15 minutes”
  • “Alert if PII detection > 0 for regulated markets”
  • “Page on-call if safety violations spike after a deployment”

Because Arize stores 1 trillion+ spans with long retention options, you can also:

• Compare safety metrics before and after model/prompt changes
• Spot slow creeps in unsafe behavior, not just spikes
• Drill from metric → trace → exact conversation → root cause

Why this matters: Without continuous monitoring, you won’t notice when a new model version subtly increases hallucinations or starts leaking PII until a customer complaint lands in your inbox.

5. Escalate: Send the right cases to human annotation queues

Humans still matter—but strategically.

Rather than reading random conversations, teams:

• Define routing rules for annotation queues:

  • All responses flagged as unsafe by guardrails
  • High-impact segments (e.g., VIP customers, regulated markets)
  • Low-confidence or ambiguous evals (where LLM judges disagree)
  • Random samples for calibration

• Use Annotation Queues in Arize to:

  • Have reviewers label: hallucination, PII, toxicity, policy violations, severity.
  • Collect better “ground truth” than what an LLM judge can infer.
  • Mark spans and traces as “golden examples” for training and testing.

• Turn annotations into datasets:

  • “Hallucination edge cases”
  • “High-risk safety violations”
  • “Correct but counterintuitive answers”
  • “Tool-call mistakes that look like hallucinations”

These datasets feed back into your CI/CD pipeline and help train or fine-tune better safety and correctness models.

Why this matters: Human review is expensive. Queues ensure the right humans look at the highest-impact events, and their work compounds into reusable assets instead of disappearing into tickets or Slack threads.

6. Close the Loop: CI/CD experiments to prevent regressions

Catching issues once is good; preventing them from reappearing is better.

With Arize AX, teams run CI/CD Experiments that:

• Use curated datasets (including annotated unsafe cases) as test suites.
• Evaluate new prompts, models, or retrieval configs using the same LLM / code evaluators used in production.
• Compare candidate vs. baseline on:
  • Hallucination rate
  • Safety / toxicity
  • PII leakage
  • Answer quality and task success
• Block or gate releases if safety metrics regress beyond thresholds.

Example policies:

• “Reject deployment if hallucination rate increases by >0.5% on safety dataset.”
• “Reject if any critical PII cases are no longer caught by guardrails.”
• “Require human sign-off on changes that impact regulated workflows.”

Why this matters: Agents are inherently non-deterministic. You can’t guarantee perfect safety, but you can ensure every change is measured against real production failures before it goes live.

Common Mistakes to Avoid

• Relying only on generic content filters or model provider flags:
  These are helpful but coarse. They often miss domain-specific risks (e.g., internal project names, customer IDs) and create a false sense of security. Layer your own code evals and LLM judges tailored to your business and policies.

• Treating incidents as one-off fires instead of training data:
  When a hallucination or unsafe output surfaces, teams will fix the prompt and move on. If you don’t log it as a trace, annotate it, and add it to a regression dataset, you’re doomed to repeat the same failure later with a different prompt or model.

Real-World Example

At my current company—a global marketplace with strict SLOs and regulated data constraints—we had an agent answering seller compliance questions. We initially sampled chats weekly and spot-checked for hallucinations and unsafe advice. It felt fine, until Legal found a few answers where the agent confidently invented policy details for a specific region.

We standardized OpenTelemetry tracing across the stack so every agent interaction logged:

• User question, retrieval context, tool calls (policy APIs), and model responses
• Model metadata (version, provider) and routing decisions
• Safety and hallucination evals per span

Then we:

1. Built LLM-as-a-Judge evaluators for:
   • “Is this answer grounded in retrieved policy documents?”
   • “Is this giving the user permission to bypass rules or misrepresent data?”
2. Added code evals for PII and internal identifiers.
3. Enabled Online Evals + Monitors in Arize with alerts on:
   • Hallucination score > threshold in compliance flows
   • Any policy violation label
4. Created annotation queues for:
   • All flagged compliance responses
   • Random 1% sample of “safe” answers for calibration

Within a week, we discovered:

• A specific routing path that skipped the policy retrieval tool under certain language conditions.
• A new model version that hallucinated more often when policy docs were long and dense.

We fixed the routing, tightened prompts, and added those failure cases as a golden dataset. Now every change to that agent is gated in CI/CD; if hallucination or policy violation rates worsen on the compliance dataset, the release is blocked.

Pro Tip: Start with a thin slice—e.g., one high-risk flow (billing, compliance, healthcare)—and instrument it fully with tracing, evals, guardrails, and annotation queues. Once that loop works end-to-end, reuse the same evaluator templates and monitors across other agents and products.

Summary

Catching hallucinations and unsafe outputs in production is not about reading more transcripts; it’s about turning every response into structured, evaluated, and monitorable data. With OTEL-based tracing, online evaluations, real-time guardrails, and targeted annotation queues, you:

• See the full flow behind every bad answer, not just the final message.
• Automatically detect hallucinations, PII leaks, and toxic content at scale.
• Convert real incidents into test datasets and CI/CD gates so regressions are caught before they ship.

Teams that treat safety as an evaluation and observability problem—not a manual QA problem—can ship agents that work under real-world constraints.

Next Step

Get Started