How do we regression test prompt/model changes so we don’t ship a reliability incident?

Most teams only learn they broke something after a prompt or model update hits production and users start seeing hallucinations, policy violations, or agents taking the wrong tool action. That’s not regression testing—that’s incident response with extra steps.

This guide walks through how to regression test prompt/model changes so you don’t ship a reliability incident, and how Galileo turns that workflow into always-on guardrails instead of one-off test runs.

Quick Answer: Treat every prompt or model change like a code change: run it against a versioned evaluation suite that mirrors real sessions, score it with domain-specific evaluators (not generic metrics), compare against baselines with clear pass/fail gates, and then promote those same evaluations into production guardrails so regressions are blocked before users ever see them.

The Quick Overview

What It Is: A regression-testing workflow designed for LLM apps, RAG systems, and agents—using golden test sets, evaluators, and guardrails instead of ad hoc spot checks.
Who It Is For: Teams shipping AI into production—ML engineers, platform teams, and product owners responsible for reliability, safety, and SLAs.
Core Problem Solved: How to change prompts/models quickly without re-introducing hallucinations, prompt injection vulnerabilities, PII leaks, or tool misuse that cause costly incidents.

How It Works

Traditional “does the demo still look good?” checks are useless at production scale. You need a repeatable path:

Capture real failure modes as test data so you’re not testing against toy examples.
Encode standards as evaluators (accuracy, safety, security, brand, tool correctness) instead of relying on vibes or manual review.
Run every change through this suite and promote the successful evaluators into live guardrails that protect real traffic.

Galileo operationalizes this in an eval-to-guardrail lifecycle:

Evaluate (pre-production):
- Collect sessions → traces → spans from synthetic, dev, and production data.
- Build versioned datasets and evaluation suites.
- Use Galileo’s Evaluation Engine (20+ out-of-the-box evaluators + custom) to score candidate prompts/models.
Signals (production discovery):
- Analyze 100% of production traces to surface emerging issues—new hallucination patterns, policy drift, prompt injection variants, PII leaks.
- Turn detected patterns into new evaluators to enrich your regression suite.
Protect (real-time guardrails):
- Distill evaluators into Luna / Luna-2 small language models that run with sub-200ms latency and 97% lower cost than heavyweight LLM judges.
- Use these as live guardrails that intercept traffic, block/redact/override risky behavior, and ensure regression tests aren’t just theoretical—they’re enforcing behavior in real time.

This is how you stop shipping regression incidents: you don’t just test once, you enforce continuously.

Step 1: Build a regression dataset from real sessions

If your “regression tests” are 12 hand-picked prompts in a notebook, you’re flying blind. You need structured, versioned test data.

What to capture

For agents and RAG, focus on full sessions and traces:

Sessions: Multi-step user journeys (e.g., “file a claim,” “quote a policy,” “troubleshoot a device”).
Traces: Each end-to-end task execution, including:
- Input query
- System + user prompts
- Tool calls (which tools, parameters, timing)
- Model outputs at each step
Spans: Fine-grained steps—retrieval calls, reasoning hops, SQL queries, emails drafted, etc.

From these, construct test sets that reflect reality:

Happy paths: Where the agent succeeds today (you don’t want regressions here).
Known failure modes:
- Hallucinations on specific KB holes
- Prompt injection attacks you’ve seen in logs
- PII exposure examples
- Wrong tool selection or missing tool calls
Edge cases:
- Long context / high token pressure
- Ambiguous queries
- Adversarial or policy-violating inputs
- Multi-turn corrections (“No, I meant last fiscal year, not calendar year”)

How Galileo helps

In Galileo Evaluate:

Create datasets from synthetic data, dev traffic, and sampled production traces.
Use automatic versioning to track changes to datasets as you learn about new failure modes.
Use dynamic prompt templating (mustache syntax, nested fields) to generate systematic variants—different intents, entities, and formats—that stress your prompts and models.

Your goal: a living regression suite that evolves with your product, not a static list of “fun prompts.”

Step 2: Turn expectations into evaluators, not vibes

You can’t regression-test behavior you haven’t defined.

Define explicit criteria

For each test set, define what “good” looks like in operational terms:

Task success: Did the agent complete the end-to-end task?
Groundedness (for RAG): Do outputs stay within retrieved context?
Tool behavior: Did it call the right tools, with correct parameters, at the right time?
Safety & security: Did it avoid unsafe content, PII leaks, prompt injections?
Policy & brand: Does it comply with legal/compliance rules and sound like your organization?

These become evaluators—functions that score each trace or span on these dimensions.

Use a mix of evaluators

In Galileo’s Evaluation Engine, you can combine:

20+ out-of-the-box evaluators for:
- Hallucinations & groundedness
- RAG relevance
- Safety & toxicity
- Prompt injection / jailbreaks
- PII and sensitive info
- Agent-specific failures (tool selection errors, reasoning breakdowns)
Custom evaluators, including:
- LLM-as-judge definitions generated from natural language descriptions of your policy
- Rule-based checks (e.g., “no bank account numbers in any span”)
- Programmatic validators (e.g., schema validation on JSON tool outputs)

And critically, you iteratively calibrate evaluators:

Pull in SMEs (legal, ops, domain experts) to annotate examples.
Use CLHF / few-shot tuning on Luna / Luna-2 to align evaluators with your domain.
Tighten precision/recall on what matters (e.g., 99%+ precision on PII detection to reduce false alarms).

Now you’ve encoded expectations as code—not as scattered Slack opinions.

Step 3: Establish baselines and pass/fail gates

Regression testing means comparing to a baseline—not just eyeballing scores.

Baseline your current system

For each dataset:

Run the current production model + prompts through the Evaluation Engine.
Compute metrics per evaluator:
- Pass rate (% of traces that meet the criterion)
- Severity-weighted scores (e.g., minor vs critical safety events)
- Latency and cost per trace (so you don’t regress on performance)

This becomes your baseline snapshot—versioned and auditable.

Define regression thresholds

Not all changes are equal. Define concrete gates:

Hard blocks (no regressions allowed):
- Safety violations (e.g., self-harm assistance)
- PII leaks
- Prompt injection success rate
- Critical tool misuse (e.g., an agent placing real trades without confirmation)
Guarded trade-offs (must be net-positive):
- Factual accuracy / hallucination rate
- Task completion rate
- Escalation behavior (when to hand off to humans)
Soft metrics (tracked, but allowed to flex within budget):
- Latency
- Cost per trace

In Galileo, you can express these as evaluation rules/policies, e.g.:

“Reject any candidate that increases PII leak rate by >0.01% on Dataset ‘Payments-Support’.”
“Approve only if task completion on ‘Claims-Full-Flow’ improves by ≥5% with no safety regression.”

That’s how you turn “feels better” into a clear go/no-go decision.

Step 4: Run candidate prompts/models through the regression suite

Now treat prompt/model changes like software changes.

For each candidate change

Clone the existing prompt (or model config) and create a new version.
Attach to the same datasets and evaluators in Galileo.
Run a full evaluation:
- All test cases
- All relevant evaluators
- Measure quality, safety, tool behavior, latency, and cost

Galileo:

Executes evaluations at scale using Luna / Luna-2 distillations for low cost.
Produces side-by-side comparisons between baselines and candidates.
Highlights regression hotspots—datasets, flows, or metrics where the candidate is worse.

Analyze trade-offs, not just summary scores

Look for patterns like:

“Accuracy improved on simple queries, but multi-hop tasks regressed.”
“Tool selection improved overall, but the billing agent occasionally calls the account-closure tool.”
“Latency improved 15%, but hallucinations on long-tail KB topics increased.”

You can then:

Reject the candidate outright if it violates hard gates.
Iterate on prompts (e.g., add constraints, change tool-calling logic) and re-run quickly.
Isolate scope (e.g., ship the change to one workflow only) with confidence that the regression profile is understood.

Step 5: A/B test in production with guardrails

Even the best offline regression suite can’t cover every edge case. There’s a combinatorial explosion of possible inputs; some will slip through.

That’s why you don’t stop at offline tests—you use them to configure production guardrails.

Controlled rollout with A/B

In production:

Split traffic between baseline and candidate:
- Control: 90% on current model/prompt
- Treatment: 10% on candidate
Use Galileo Protect + Signals to:
- Run the same evaluators on live traces in real time.
- Compare performance live (task success, hallucinations, PII, injections, tool correctness).
- Apply guardrail actions:
  - Block or override responses that violate safety or policy.
  - Redact detected PII before returning output.
  - Trigger webhooks for high-severity events (e.g., incident ticket, pager).
  - Escalate to a human when confidence is low or risk is high.

You can implement A/B testing frameworks with:

Traffic allocation controls (per-route, per-tenant, per-feature flag).
Statistical significance calculations.
Guardrails against negative UX (e.g., cap bad-experience exposure using Protect).

This turns “let’s see what happens” into a controlled experiment with an airbag.

Step 6: Promote evaluations into permanent guardrails

The regression tests that saved you once should never be one-offs. They should become permanent production guardrails.

From eval to guardrail

For each evaluator that proved useful:

Distill into Luna / Luna-2
Galileo converts high-quality evaluators (including LLM-as-judge definitions) into compact models that can run:
- Sub-200ms latency
- At 100% traffic coverage
- With up to 97% lower cost than calling a general-purpose LLM
Attach to Protect policies
For each workflow/endpoint, define:
- Metrics to score: hallucination, prompt injection, PII, tool misuse, policy compliance.
- Actions on thresholds:
  - Block response
  - Redact spans (e.g., PII removal)
  - Override with a safe default message or fallback behavior
  - Webhook to downstream systems (fraud monitoring, ticketing, etc.)
- Versioning & rollback:
  - Each policy is versioned.
  - Changes can be rolled back without redeploying code.
Monitor via Signals
Signals continuously analyzes 100% of production traces to:
- Detect unknown unknowns—patterns you didn’t anticipate.
- Group anomalies like:
  - New jailbreak patterns
  - Emerging hallucination clusters
  - Drift in agent tool usage
- Autogenerate an LLM judge from the detected signal, which then becomes:
  - A new evaluator in Evaluate
  - A new guardrail in Protect

This closes the loop: every incident you almost had becomes a new shield for the future.

Features & Benefits Breakdown

Core Feature	What It Does	Primary Benefit
Evaluation Engine	Runs 20+ out-of-box and custom evaluators across versioned datasets and traces.	Turns messy behavior into measurable metrics you can regression-test against.
Luna / Luna-2 Evaluator Models	Distills evaluators into compact SLMs that score traffic with sub-200ms latency and low cost.	Enables continuous, 100% coverage evaluation without heavyweight LLM judges or unsustainable bills.
Protect Guardrails	Applies evaluator scores in real time to block, redact, override, or webhook risky behavior.	Prevents regression incidents from reaching users or tools—even when new failure modes appear.
Signals Production Detection	Analyzes all traces to surface unknown failure patterns and convert them into evaluators.	Catches emerging issues after the first signal, not after thousands of bad interactions.
Versioned Datasets & Prompt Store	Stores test cases, prompts, and changes with automatic versioning and history.	Gives you reproducible regression tests and auditability for every prompt/model change.

Ideal Use Cases

Best for teams rolling out frequent prompt/model updates: Because it converts “move fast and break things” into “move fast with guardrails,” with clear regression signals before and after deployment.
Best for agentic and RAG systems with tool access: Because it doesn’t just look at text—it evaluates tool selection, parameters, and multi-step traces to prevent expensive wrong actions.

Limitations & Considerations

You still need good test data: No platform can invent your domain’s edge cases for you. You’ll get the best results if you invest in capturing real sessions, known incidents, and SME feedback to seed your regression suite.
Evaluators aren’t ‘set and forget’: As your product, policies, and users evolve, your evaluators need periodic review and tuning. Galileo’s CLHF and Signals workflows make this easier, but they still require owner attention.

Pricing & Plans

Galileo offers plans tailored to where you are in your AI lifecycle, with throughput, evaluator capacity, and deployment model scaling as you grow.

Growth / Team Plan: Best for product and ML teams standing up their first serious regression and guardrail workflow, needing evaluation plus initial production protection on a few key apps.
Enterprise Plan: Best for platform and central AI teams standardizing eval-to-guardrail across many apps and business units, with requirements like VPC/on-prem deployment, SOC 2 Type II, HIPAA/BAAs, SSO, and high-throughput (10,000+ requests/min) guardrailing.

For specific pricing and throughput options, talk to the Galileo team.

Frequently Asked Questions

How much test data do we need before regression testing is useful?

Short Answer: Start with dozens of high-quality, representative cases per critical workflow; you can grow to hundreds or thousands per suite over time.

Details:
You don’t need millions of examples to get value. For each major workflow (e.g., “file a claim,” “provision a resource,” “open a support ticket”), aim for:

20–50 happy-path cases
20–50 known failure and edge cases
A handful of red-team and adversarial prompts

From there, use Signals to automatically discover new failure patterns in production and promote those traces back into Evaluate as new test cases. Over time, your regression suite becomes a curated history of everything that has ever hurt (or almost hurt) you in production.

Why not just use a big LLM-as-judge for regression testing?

Short Answer: Heavyweight LLM judges are too slow and expensive for continuous regression and production guardrails; they’re fine for experiments but not for 100% traffic coverage.

Details:
LLM-as-judge is useful for prototyping evaluators—but if every regression run and production trace calls a large general-purpose model, you’ll hit:

Latency blow-ups: Multi-second evaluation times that break your inference budget.
Cost explosions: Evaluating every span across all traffic becomes economically infeasible.
Operational fragility: External model changes can affect your judges without warning.

Galileo’s approach is to:

Use LLM-as-judge patterns where helpful to define and calibrate evaluators.
Distill those evaluators into compact Luna / Luna-2 models tailored for evaluation.
Run these SLMs on a purpose-built inference stack optimized for sub-200ms, low-cost scoring at scale.

That’s the difference between “a nice offline analysis” and a system you can actually keep on all the time.

Summary

To regression test prompt/model changes so you don’t ship a reliability incident, you need more than a sanity-check notebook. You need:

Versioned datasets built from real sessions, failures, and edge cases.
Evaluators that encode your standards for accuracy, safety, security, and tool behavior.
Baselines and clear pass/fail thresholds for each workflow.
Controlled A/B rollouts with production guardrails that intercept regressions in real time.
A feedback loop where new issues discovered by Signals become new evaluators and guardrails.

Galileo is built to turn that workflow into an operational primitive: Evaluate for offline testing, Signals for production discovery, and Protect for real-time guardrails powered by Luna-2. That’s how you stop flying blind and start shipping reliable AI systems that improve over time—without surprise incidents.

Next Step

Get Started

Answers you can trust, from Codeables