How do I set up repeatable evals for an LLM app so prompt changes don’t ship regressions?

Most teams learn the hard way that LLM apps fail in production not because the “big model changed,” but because a seemingly harmless prompt tweak shipped a regression. If you don’t have repeatable evals in place, every prompt change is a gamble—especially once you’re dealing with multi-step agents, tool calls, and retrieval.

Quick Answer: Set up repeatable evals by standardizing traces for every LLM and tool call, curating a stable evaluation dataset, and wiring both offline and online evaluations into CI/CD so no prompt or routing change ships without passing checks. Use a mix of LLM-as-a-Judge, code-based evals, and human annotation queues, then continuously feed production traces back into your eval sets to catch regressions early.

Why This Matters

LLM apps are non-deterministic and deeply sensitive to prompts, tools, and retrieval strategies. Without a repeatable evaluation loop, you can’t tell if a change actually improved quality, quietly broke an edge case, or just shifted error modes. For teams with SLAs, compliance requirements, or real cost exposure, this isn’t a “nice-to-have”—it’s the difference between a polished demo and a production system you can trust.

Key Benefits:

• Detect regressions before users do: Evaluation-driven CI/CD lets you block deployments when quality drops, instead of finding out when tickets spike or NPS drops.
• Iterate faster with guardrails: When every prompt, tool, or routing change is evaluated the same way, you can experiment aggressively without fear of breaking core flows.
• Turn production chaos into signal: With good tracing and annotation queues, you can convert real-world failures into golden datasets that continuously improve your agents.

Core Concepts & Key Points

Concept	Definition	Why it's important
Evaluation Dataset	A curated set of representative inputs, expected behaviors, and scoring criteria for your LLM app or agent.	Forms the baseline you use to compare prompt and routing changes, so you can detect regressions instead of eyeballing examples.
LLM-as-a-Judge Evals	Using an LLM to score outputs (e.g., correctness, hallucination, tool selection) based on structured guidelines and rubrics.	Covers large test surfaces you’ll never fully label by hand and gives you fast, repeatable signals on every sub-call.
Online + Offline Evals	Offline evals run during development and CI; online evals run on live traffic traces in production.	Offline evals catch issues before deployment; online evals catch long-tail failures and drift as user behavior and data change.

How It Works (Step-by-Step)

The most reliable way I’ve seen to avoid shipping prompt regressions is to treat evals like unit tests, integration tests, and canary checks for your agents—backed by full tracing. At a high level, you:

1. Instrument everything with open-standard tracing.
2. Build evaluation datasets from real traces.
3. Wire evaluations into a CI/CD + monitoring loop.

Below is how I’d set it up, using Arize AX/Phoenix as the backbone but keeping everything compatible with open standards like OpenTelemetry and OpenInference.

1. Instrument Every LLM and Tool Call

One platform. One trace per request. Every span matters.

Goal: For any user request, you can reconstruct the full flow: prompts, tools, retrieval, intermediate reasoning, and final answer. Without this, evals will always be partial and fragile.

What to do:

• Adopt OpenTelemetry + OpenInference conventions

  • Use OTEL for spans/traces and OpenInference for LLM-specific attributes (prompt, completion, model, temperature, tool names, etc.).
  • Instrument:
    • LLM calls (system/user/assistant messages, parameters, tokens).
    • Tool calls (inputs, outputs, latency, status).
    • Retrieval steps (query, top-k, source IDs, scores).
    • Router decisions and multi-agent handoffs (which agent, why, what context).

• Use Phoenix or AX for tracing & visualization

  • Send spans to Arize Phoenix (self-hosted) or Arize AX.
  • Validate that each user session becomes a trace with a multi-step graph:
    • Root span: user request.
    • Child spans: RAG retrieval, tools, LLM subcalls, validation steps, etc.
  • Confirm you can click into any LLM span and see the exact prompt, model, and output for replay.

Why this matters for evals:

• You can attach eval scores directly to spans (e.g., “hallucination_score” on an answer span or “tool_call_correctness” on a tool span).
• You can rebuild eval datasets directly from real production traces—no manual log scraping.
• You can distinguish between “model was wrong” and “tool or retriever gave bad data.”

2. Define Success Metrics for Your LLM App

You can’t prevent regressions if you can’t define what “good” is.

Start with a minimal set of metrics tied to your use case:

• Question answering / RAG

  • Relevance / answer correctness
  • Hallucination rate (does it invent facts outside provided context?)
  • Context utilization (does the answer actually use retrieved docs?)
  • Latency and cost per request

• Customer support / workflows

  • Task completion (was the right action taken?)
  • Policy/guardrail adherence
  • Escalation correctness (did it hand off when it should?)

• Tool-using agents

  • Tool selection correctness (was the right tool chosen?)
  • Argument/parameter extraction correctness
  • Path convergence (did the agent reach the right final state even if it explored multiple paths?)

Each of these becomes a dimension in your eval rubric—something you’ll score via LLM-as-a-Judge, code evals, or humans.

3. Build a Repeatable Evaluation Dataset

You need a stable, versioned evaluation dataset that represents your critical flows and edge cases. It doesn’t have to be huge; it has to be representative.

How to build it:

1. Seed from production traces

   • Use Phoenix/AX to search for:
     • High-traffic intents.
     • Known failure clusters (e.g., “hallucination,” “wrong tool,” “missed step”).
     • Critical user segments (e.g., VIPs, regulated regions).
   • Export these traces into a dataset with:
     • Input (user query + context like account type).
     • Expected behavior (description or target answer).
     • Metadata (intent, slice, region, model used).

2. Add golden examples with strong labels

   • For each important path, add:
     • A “gold” output (the best response you’d accept).
     • One or more counterexamples (bad responses you want to flag).
   • Use human annotation queues (AX) to collect ratings from internal SMEs or support teams.

3. Version and slice your datasets

   • Maintain versions: v1-baseline, v2-complaints, v3-long-tail, etc.
   • Tag slices: by language, product type, complexity, or risk level.
   • Use these slices to run targeted experiments later (e.g., “Did this prompt change hurt compliance queries?”).

This evaluation dataset becomes the contract your prompts must satisfy before shipping.

4. Define LLM-as-a-Judge and Code-Based Evals

You almost never have labels for more than a tiny fraction of your I/O pairs. That’s where LLM-assisted evals come in—as long as they’re structured and repeatable.

Use a layered eval strategy:

a) LLM-as-a-Judge (semantic and behavioral checks)

Use a model (not necessarily your production model) to score outputs with detailed instructions and examples. In Arize, these are just evaluators you attach to spans.

Example rubrics:

• Answer correctness (0–5)
  “Given the question, the retrieved context, and the answer, score how correct and helpful the answer is. Penalize hallucinations, missing key facts, or contradictions.”

• Hallucination detection (binary or 0–1)
  “Does the answer rely only on the provided context? If it introduces information not present or implied, mark as hallucinated.”

• Tool selection accuracy
  “Given the user request and the tools available, did the agent choose the minimal set of appropriate tools? Did it skip tools that were required?”

• Parameter/field extraction
  “Given the source text and the extracted JSON, is each field correct, complete, and aligned with the instructions?”

Best practices:

• Use open, configurable eval templates—not a black-box scoring API.
• Anchor judgments in visible context (prompt, retrieved docs, tool schema) so you can debug disagreements.
• Log the judge’s explanations alongside scores so humans can review borderline cases.

b) Code-based / deterministic evals

Anywhere behavior can be formalized, use code to score outputs:

• JSON schema validation
• Regex or parsing checks
• Business rule checks (e.g., “discount <= 30%”)
• For numeric or structured answers, direct equality or tolerance checks

These become hard guardrails: if they fail in offline evals, the build fails.

c) Human annotation for high-risk flows

For regulated or high-impact flows (e.g., compliance guidance, medical-ish content):

• Route eval failures or low-confidence LLM-judge cases into annotation queues.
• Have SMEs rate correctness, risk, and usability.
• Feed these back into your golden datasets and improve your LLM-as-a-Judge prompts.

5. Wire Evals into CI/CD: No Prompt Change Ships Unchecked

This is the step most teams skip—and where regressions sneak in.

Use Arize AX CI/CD Experiments (or your CI system calling AX APIs) to enforce:

1. Treat prompt changes like code changes

   • Store prompts in version control (Git, config repo, prompt hub in AX).
   • Every change triggers a pipeline:
     • Replay your evaluation dataset through both the current and candidate prompt/model.
     • Run all evals (LLM-as-a-Judge + code + any existing human labels).
     • Compare metrics and slices.

2. Define clear regression thresholds

   • E.g., a candidate is rejected if:
     • Overall correctness score drops by >1% on any critical slice.
     • Hallucination rate increases by >0.5% on compliance queries.
     • Any hard guardrail (schema, policy) fails on previously passing examples.

3. Use experiments to test multiple variants

   • Configure experiments in AX to compare:
     • Prompt A vs Prompt B.
     • Old routing strategy vs new tool chain.
     • Different temperature or model choices.
   • Pick winners based on evaluation metrics, not subjective sampling.

4. Make failures visible

   • For a failed experiment:
     • Show per-span diffs: old vs new answers, scores, and traces.
     • Highlight which slices regressed and why.
   • This turns “the build failed” into “we know exactly which examples regressed.”

If you’re not using AX directly, mimic this pattern: your CI service calls a runner that replays the dataset, runs evals, and enforces thresholds before deploying.

6. Add Online Evals to Catch Drift and Long-Tail Failures

Offline evals prevent known regressions; online evals protect you from reality changing underneath you.

With Arize AX:

• Attach evaluators to production spans

  • Every time your app answers a user, run lightweight LLM-as-a-Judge evals:
    • Was it correct?
    • Did it hallucinate?
    • Did it use context?
  • Store these scores as span attributes.

• Build dashboards and alerts

  • Track metrics across:
    • Time (did quality regress after a new model release?)
    • Slices (language, product, region, model version).
  • Set alerts for:
    • Sudden spikes in hallucination rate.
    • Drops in task completion.
    • Latency or cost outliers.

• Feed online evals back into your datasets

  • When you see a pattern of failures in a slice, push those traces into:
    • A new evaluation dataset version.
    • An annotation queue for SMEs.
  • This keeps your offline evals aligned with real-world usage.

This is how you close the loop between AI development and production instead of running static experiments that grow stale.

7. Make Prompts and Agents Self-Improving (Optional, but Powerful)

Once you trust your evals, you can start using them as optimization signals:

• Prompt optimization loops

  • Use evaluations + annotations to propose prompt edits and test them in experiments.
  • In Arize AX, you can replay and tweak prompts directly from spans in a playground, then promote the best variant to production via CI.

• Self-correction in the agent

  • If an eval (or heuristic) detects a likely hallucination or missing step, the agent can:
    • Re-query a tool.
    • Ask for more context.
    • Re-run a step with different parameters.
  • Importantly, you still evaluate these self-corrections—they’re not magic; they’re just another path in your trace.

Common Mistakes to Avoid

• Treating evals as a one-time project instead of a loop

  • How to avoid it: Version your datasets, keep adding production edge cases, and schedule periodic evaluation runs even when you’re not actively changing prompts.

• Relying only on “spot checks” or product reviews

  • How to avoid it: Implement automated CI/CD Experiments and Online Evals so every change is tested against the same criteria, not just a handful of eyeballed examples.

• Using opaque, black-box eval services with no trace context

  • How to avoid it: Keep your evals grounded in open templates and attach them to spans with full prompt/context visibility, so you can debug and improve them over time.

• Ignoring tool and retrieval errors in favor of blaming the model

  • How to avoid it: Evaluate every sub-call—retriever outputs, tool calls, and routing decisions—not just the final answer span.

Real-World Example

At my current org, we had a multi-agent support assistant that looked great in demos but kept misrouting “refund + fraud” queries in production. The root cause wasn’t obviously visible in logs: sometimes the agent called the “RefundPolicy” tool, sometimes “FraudInvestigation,” and sometimes both. We only noticed when escalation tickets spiked.

We standardized OTEL tracing with OpenInference attributes and pushed spans into Arize AX. From there:

1. We built an evaluation dataset from actual misrouted traces and golden “ideal” flows.
2. We defined LLM-as-a-Judge evals for:
   • “Did the agent pick the right tool chain?”
   • “Did the final answer follow company policy?”
3. We added a code-based eval to verify that if fraud_flag=true, the FraudInvestigation tool must be called before IssueRefund.
4. We wired CI/CD Experiments so any change to the routing prompt or tools had to outperform the baseline on those evals.

Within a week, two things happened:

• A new routing prompt that “felt better” in manual tests was blocked in CI because it regressed fraud handling on a small but critical slice.
• We discovered that some “wrong tool” errors were actually retriever issues, not model issues, by inspecting the full multi-agent traces.

Today, that same system ships prompt or tool changes frequently, but our fraud-related regressions have effectively vanished because they simply can’t pass the gating evals.

Pro Tip: Start small: pick your top 50–100 critical traces, create a simple eval rubric (correctness + hallucination + tool usage), and wire just that into CI. Once you see a regression get caught before it hits production, it becomes much easier to justify expanding the eval surface.

Summary

Repeatable evals for an LLM app aren’t about a single magic score—they’re about building a loop:

• Trace everything with open standards so you can see the full flow.
• Curate evaluation datasets from real traces and golden examples.
• Use layered evals—LLM-as-a-Judge, code checks, and human annotations—to measure correctness, hallucinations, and tool usage.
• Integrate offline evals into CI/CD so prompt and routing changes can’t ship regressions.
• Run online evals in production to catch drift and feed new edge cases back into your datasets.

If you can’t trace every step, evaluate every sub-call, and connect online behavior back into experiments, you’re still in demo mode. Once you can, prompt changes stop being scary—and become the fastest way to ship agents that actually work.

Next Step

Get Started