How do I set up online evals in Arize to flag hallucinations, policy violations, or PII leakage in production?

Online evals are how you stop “nice demo” agents from quietly leaking PII, hallucinating, or slipping past policy once they hit real traffic. Instead of waiting for a weekly log review, Arize can evaluate every (or sampled) production response in real time, score it for risk, and trigger alerts or guardrails when something goes wrong.

Quick Answer: Use Arize’s Online Evals to stream your production traces (via OTEL/OpenInference), attach one or more evaluators (LLM-as-a-judge, regex/code checks, or your own functions), and configure thresholds + alerts for hallucinations, policy violations, and PII leakage. Start with a few high-signal eval templates, run them on live spans, and wire the results into dashboards, alerts, and CI/CD experiments so bad behavior is caught and blocked before it hits users.

Why This Matters

In production, non-deterministic LLMs and agents will eventually hallucinate, mishandle sensitive data, or break policy—especially as prompts, tools, and models change. Without online evals, you only see these failures after the fact, when users complain or an audit lands on your desk.

Online evals in Arize close that gap. You continuously score live traffic for hallucinations, PII exposure, and policy violations, then use those signals to:

• alert on regressions,
• throttle or block risky responses,
• and feed edge cases into your evaluation loop.

Key Benefits:

• Catch bad outputs instantly: Real-time scoring on every (or sampled) response means hallucinations, PII leaks, and policy violations are detected as they happen—not days later.
• Guardrails you can measure: Instead of brittle regex-only filters, you combine LLM-as-a-judge evals, code checks, and annotation queues to enforce policies with clear metrics and thresholds.
• Close the loop between prod and dev: Online eval results flow back into Arize datasets and experiments, so you can A/B new prompts/models and gate releases on quality and safety, not gut feel.

Core Concepts & Key Points

Concept	Definition	Why it's important
Online Evals	Evaluations that run automatically on live production spans as they’re ingested into Arize AX, producing scores/labels in real time.	Turns observability into an active safety layer: you can alert, dashboard, and gate on quality and safety metrics continuously.
LLM-as-a-Judge Evaluators	Eval prompts that use an LLM to label behavior (e.g., “Does this output contain PII?”) based on your policy rubric.	Captures nuanced issues (subtle hallucinations, policy context, borderline PII) that simple pattern matching misses.
Guardrail Metrics & Alerts	Thresholded metrics (e.g., pii_violation_rate > 0.1%) with notifications and routing logic.	Translates qualitative risk into numbers that can drive SLOs, alerts, and CI/CD gates for prompt/model changes.

How It Works (Step-by-Step)

At a high level, you:

1. Instrument your app with OpenTelemetry/OpenInference so Arize receives full traces and outputs.
2. Define evaluators for hallucination, policy violation, and PII leakage.
3. Configure Online Evals to run those evaluators on live spans and set up alerts/guardrails.

Below is the pattern I recommend for teams running regulated workloads.

1. Instrument production with OTEL and OpenInference

One platform. One trace for every request.

Arize works best when every agent call, tool call, and model response becomes a span in a trace. That’s what your online evals will run on.

Implementation outline:

• Add OTEL SDK to your service(s) or gateway (Python/Node/Go/Java all work).
• Adopt OpenInference conventions so model I/O is clearly labeled:
  • input_text / output_text (or structured fields for multi-part messages),
  • model name, temperature, top_k, etc.,
  • tool call metadata (tool name, arguments, status).
• Group spans into:
  • a session (end-user conversation or workflow), and
  • a trace (single request path through agents/tools).

At minimum, each LLM response span should include:

• User prompt / agent context (sanitized as appropriate),
• Model output text,
• Any tool outputs used by the model,
• Metadata: model provider, latency, token usage, request ID.

Once you send these spans to Arize AX or Phoenix (self-hosted), you’ll see them as traces and can start layering online evals on top.

2. Define your online evals (hallucination, policy, PII)

Think in terms of risk categories, not just individual checks. For this use case, I usually define three buckets:

1. Hallucination / factuality risk
2. Policy violations (safety, compliance, brand)
3. PII detection and leakage

Each bucket can combine:

• LLM-as-a-judge evals (for nuance),
• Code/regex evals (for hard constraints),
• Optional human annotation queues (for ambiguous edge cases).

2.1 Hallucination detection evals

Factuality is tricky because you need a reference. Common strategies:

• For RAG:

  • Pass both the retrieved context and LLM answer into a judge model.
  • Ask it to score: “Is this answer fully supported, partially supported, or unsupported by the context?”
  • Output: categorical label (e.g., supported, partially_supported, hallucinated) plus a confidence/score.

• For transactional agents/tools:

  • Compare tool outputs vs. final answer:
    • Did the agent ignore the tool result?
    • Did it invent extra fields or entities?
  • Output: boolean or score, e.g., tool_consistency_score.

In Arize, you define this as an LLM eval template, then attach it as an Online Eval on the relevant spans (e.g., the final answer span of a user-facing agent).

2.2 Policy violation evals

Policy violations cover things like:

• Harassment, hate, self-harm encouragement,
• Disallowed topics (e.g., medical diagnosis, financial advice for certain users),
• Regulatory rules (e.g., “never give legal advice,” “don’t answer questions about minors”).

Here I recommend a layered approach:

• LLM-as-a-judge policy rubric:

  • Prompt a judge model with:
    • The user request,
    • The model response,
    • Your policy document (or a summarized rubric).
  • Ask it to classify the response into categories, e.g., clean, borderline, violation, plus a short explanation.
  • Output: policy_violation_category, policy_violation_score, and explanation.

• Hard-coded rule evals:

  • Regex/keyword checks for specific forbidden phrases, URLs, or patterns.
  • Geo/jurisdiction-specific rules (e.g., “if user_country == ‘DE’ and topic == ‘X’ → violation”).

These become separate evaluators in Arize; Online Evals can run both on the same span and you can combine their outputs into a composite metric if needed.

2.3 PII leakage evals

PII is where regulators and auditors look first.

Combine:

• Pattern/regex-based detectors for:
  • Emails, phone numbers, SSNs, credit card numbers,
  • IBANs, government IDs, etc.
• LLM-as-a-judge PII inspector:
  • Prompt: “Does this text contain personal data about an identifiable individual? If yes, what type (name, address, etc.) and is it sensitive under [regulation]?”
  • Include your policy for what counts as PII vs. allowed context.

For production, I typically:

• Use regex/code eval as a hard fail (very low false positives),
• Use the LLM PII eval as a sensitivity layer (good at spotting free-form PII but may need threshold tuning).

3. Configure Online Evals in Arize

Once your evaluators exist, you wire them into Online Evals so they run automatically on live traffic.

The flow in Arize AX typically looks like this:

1. Select the spans to evaluate

   • Target the specific span type that carries the user-facing output, e.g., span.kind = "llm" AND span.attributes["role"] = "assistant_final".
   • Optionally filter by environment (prod vs. staging), model name, or route.

2. Attach evaluators

   • Choose the hallucination, policy, and PII evaluators you defined.
   • Configure:
     • Sampling rate (e.g., 100% for launch, then 5–20% once stable),
     • Max concurrency / rate limits to control eval token spend.

3. Define evaluation outputs

   • For each evaluator, standardize outputs like:
     • hallucination_label, hallucination_score,
     • policy_violation_label, policy_violation_score,
     • pii_violation_label, pii_violation_score.
   • Store these as span-level attributes in Arize so they’re first-class fields for dashboards, filters, and alerts.

4. Turn on Online Evals

   • Enable the job so every new qualifying span gets evaluated in real time.
   • Watch the initial metrics for a day to tune thresholds and catch any misconfigured prompts or regex rules.

4. Create dashboards and alerts for guardrail metrics

Online evals turn raw responses into metrics you can manage. Next, you define what “healthy” looks like.

In Arize, you can:

• Create metrics such as:
  • hallucination_rate = count(hallucination_label == "hallucinated") / total_spans
  • policy_violation_rate = count(policy_violation_label == "violation") / total_spans
  • pii_violation_rate = count(pii_violation_label == "violation") / total_spans
• Slice by:
  • Route / feature (e.g., “support agent” vs. “search assistant”),
  • Model version / prompt version,
  • User segment or jurisdiction (where allowed by your policies).

Set up alerts on these metrics:

• Example thresholds:
  • hallucination_rate > 1% for 5 minutes → page on-call, auto-rollback candidate.
  • pii_violation_rate > 0 in any regulated region → immediately throttle or block that route.
  • Sudden spike in policy_violation_rate → hold new deployment in CI/CD pipeline.

This is where online evals stop being “nice metrics” and become operational guardrails.

5. Wire Online Evals into CI/CD and experiments

You don’t just want to spot problems—you want to prevent regressions before they hit prod.

With Arize:

• Experiments:

  • Define A/B tests comparing:
    • New vs. old prompts,
    • Model versions or providers,
    • Agent routing strategies.
  • Use your online eval metrics (hallucination_rate, pii_violation_rate, policy_violation_rate) as primary or secondary experiment metrics.

• CI/CD gates:

  • Before shipping a change, run it on:
    • A curated offline dataset (using offline evals),
    • A small percentage of online traffic (canary deployment with Online Evals).
  • Block rollout if:
    • Hallucination or PII metrics regress beyond your budget,
    • Policy violations cross your defined thresholds.

This is how teams like Booking and Siemens use evaluation-driven CI/CD to avoid silent regressions as they iterate on prompts and agent logic.

6. Use annotation queues to refine your evals

LLM judges and regex rules are a strong start, but you’ll hit gray areas:

• “Is this PII or just a public job title?”
• “Is this advice truly medical advice, or general wellness guidance?”
• “Is this partially supported answer acceptable for this use case?”

For those, use Human Annotation and Queues:

• Route borderline cases:
  • For example, pii_violation_score between 0.3 and 0.7, or policy_violation_label = "borderline".
• Have human reviewers:
  • Confirm whether it’s a violation,
  • Label violation type,
  • Optionally supply a corrected response.

Arize turns these annotations into a golden dataset:

• You refine your LLM-judge prompts to better match human labels.
• You create additional code-based evals for recurring patterns.
• You use this dataset in experiments when testing new models or prompts.

Over time, this feedback loop significantly reduces both false positives and false negatives in your online evals.

Common Mistakes to Avoid

• Treating Online Evals as “nice-to-have” metrics instead of SLOs:

  • If hallucination/PII rates aren’t tied to alerts, error budgets, or deployment gates, they’ll be ignored when things get busy.
  • Fix: Define explicit SLOs (e.g., hallucination_rate < 0.5%, pii_violation_rate = 0) and log them in your incident runbook.

• Using only regex filters without LLM or human judgment:

  • Pattern-based checks miss free-form PII and nuanced policy violations; they also create brittle, noisy rules.
  • Fix: Combine regex/heuristics with LLM-as-a-judge evaluators and periodically calibrate them using annotation queues.

Real-World Example

At my last org, we rolled out a support agent that summarized account information for users in the EU. The agent was grounded on an internal RAG system, and we thought we were safe because we had basic regex checks for obvious PII patterns.

Once we turned on Online Evals in Arize:

• A PII LLM-judge eval started flagging responses where the agent casually mentioned “your street name and city” in summaries. No explicit address string matched our regex, but the combination still violated our internal policy.
• A policy eval revealed that a new prompt tweak caused the agent to give borderline individualized financial suggestions in a specific market segment.
• Because we had alerts wired to these evals, the spikes showed up in our dashboards within minutes of the deployment. We:
  • Rolled back the new prompt,
  • Tightened our “allowed fields” constraints in the retrieval layer,
  • Updated our eval prompts and regex checks based on the flagged examples.

Those same flagged spans became a golden dataset for later experiments, letting us re-test future prompt changes against the exact types of failures we’d seen in production.

Pro Tip: Start with a narrow scope: one route, three evaluators (hallucination, policy, PII), and clear thresholds. Once that pipeline is stable—traces → online evals → alerts → experiments—copy the pattern to the rest of your agents instead of custom-building from scratch each time.

Summary

Online Evals in Arize turn your production agents from a black box into a monitored, measurable system with real guardrails. By instrumenting your app with OTEL/OpenInference, defining LLM-as-a-judge and code-based evaluators for hallucinations, policy violations, and PII leakage, and wiring those into Online Evals with alerts and experiments, you:

• detect risky behavior in real time,
• enforce safety and compliance with measurable thresholds,
• and feed production edge cases back into your evaluation datasets and CI/CD pipeline.

That’s how you ship agents that actually work in production—not just in demos.

Next Step

Get Started