What’s the best way to build and maintain a golden test set for a RAG app as docs and policies change?

Most RAG teams don’t realize their “golden test set” is stale until it’s too late—after a policy change ships, the model keeps citing old rules, or a pricing update doesn’t show up in answers. If your docs and policies move fast but your eval set doesn’t, you’re not testing your real system anymore—you’re testing last quarter’s.

This is where a disciplined approach to building and maintaining a golden test set matters. For a RAG app that sits on top of live, evolving documentation, you need more than a one-off benchmark. You need a living evaluation asset that tracks doc changes, policy drift, and new failure modes—then turns into production guardrails.

Below is a concrete, practitioner-style playbook for doing exactly that.

The Quick Overview

What It Is: A golden test set for a RAG app is a curated, versioned collection of queries, docs, and expected answers that represent high-value, high-risk, and edge-case scenarios for your domain—used to measure and govern system behavior over time.
Who It Is For: Teams shipping RAG-backed assistants, internal knowledge tools, or customer-facing agents who care about policy adherence, factuality, and traceable reasoning—not just “okay” answers.
Core Problem Solved: It prevents silent regressions and policy drift as your documentation, models, and prompts change—so you catch errors in a controlled eval loop instead of after customers or auditors do.

How It Works

A robust golden test set for a RAG app follows a lifecycle, not a single setup step:

Define & scope: Decide what “good” looks like in your domain and identify the most critical tasks, policies, and failure modes.
Collect & curate: Build an initial dataset from real interactions, synthetic edge cases, and SME-reviewed answers with explicit rationales.
Version & evolve: Keep your test set, prompts, and underlying documents in lockstep as policies change—and continuously promote new production patterns and incidents into the golden set.

A platform like Galileo operationalizes this lifecycle:

Evaluate turns queries + docs + expected answers into structured test suites using the Evaluation Engine (with 20+ RAG-specific evaluators).
Signals watches 100% of production traces to surface new failure patterns and candidate test cases you didn’t think to write.
Protect distills your best evaluators into Luna / Luna‑2 models and runs them in real time as guardrails, so the lessons from your golden set actively intercept bad outputs.

Let’s break down the mechanics.

Step 1: Define what “golden” means for your RAG app

Before you write a single test, you need to answer: “Golden with respect to what?”

1. Identify critical workflows

List the top 10–20 workflows where being wrong is expensive:

Pricing, discounts, and contract terms
Compliance and legal guidance
Security and data-handling policies
HR, benefits, and internal policy questions
Support workflows for high-impact customers or products

These become the backbone of your golden set. For each workflow, capture:

User intent: What the user is trying to accomplish
Business risk: What happens if the answer is wrong, incomplete, or outdated
Required guardrails: Must-cite docs, forbidden content, escalation rules

2. Define your evaluation criteria

A golden test set isn’t just “answer looks good.” You need measurable dimensions:

For RAG specifically, think in terms of:

Groundedness: Is every key fact supported by retrieved sources?
Retrieval quality: Are the right docs pulled (coverage and relevance)?
Policy adherence: Does the answer follow current internal or regulatory rules?
Completeness: Does it answer the whole question, including edge clauses?
Safety/compliance: No PII leaks, no restricted advice, no prohibited claims.

In Galileo, these map directly to evaluators in the Evaluation Engine—for example:

RAG hallucination / groundedness checks
RAG retrieval coverage and over-/under-fetching
Safety & security evaluators (PII, prompt injection, policy violations)
Custom domain evaluators generated from natural language descriptions

Write these criteria down as a short “eval spec”: a single doc that says:

“For pricing queries, answers must: (1) cite the current pricing policy doc, (2) not mention deprecated tiers, (3) include the ‘subject to change’ disclaimer, and (4) avoid hard commitments beyond documented SLAs.”

That spec is the contract your golden set enforces.

Step 2: Build your initial golden test set

Once you know what to measure, you can build a dataset that actually stresses your system.

1. Start from real interactions

Don’t invent everything from scratch. Use:

Logged tickets and emails
Chat transcripts from current systems
Top search queries from your help center or docs site

For each, normalize into a consistent structure:

Query / user message
Context docs that should be retrieved (links or IDs)
Expected answer (gold response)
Rationale / notes (why this answer is correct, what policy it reflects)
Tags: policy area, product area, risk level, customer tier, etc.

This is where a dataset collection system—like Galileo’s dataset collections—helps: you can structure, organize, and auto-version these test cases instead of keeping them in scattered spreadsheets.

2. Augment with synthetic edge cases

Real logs show the “center” of user behavior. Your golden set must also cover the edges:

Ambiguous questions that require clarifying assumptions
Multi-step questions where requirements change mid-conversation
Adversarial phrasing (“Are there any exceptions where I can share customer PII?”)
Policy boundary questions (“What’s the max discount I can offer without approval?”)

You can generate synthetic tests by:

Using an LLM to mutate real queries (change dates, products, jurisdictions)
Prompting an LLM to produce “tricky” versions of critical questions
Having SMEs propose the top 10 “most abused” or “most misunderstood” policy scenarios

From the Galileo knowledge base: start by collecting actual user interactions, then augment with synthetic examples designed to challenge your agent’s capabilities—especially adversarial and ambiguous instructions and scenarios where users change their minds mid-conversation.

3. Involve subject matter experts (SMEs)

Your golden set is only as good as its labels. For policy- and compliance-sensitive RAG apps, you need SMEs to:

Approve or edit expected answers
Flag required citations and authoritative docs
Annotate “hard fails” (e.g., “If the answer omits this clause, it’s unacceptable”)

In Galileo, SME annotations become training data for evaluators:

You can transform this labeled feedback into CLHF-tuned evaluators.
Over time, these evaluators get distilled into Luna‑2 models that capture your domain’s definition of “correct”—not some generic benchmark.

Step 3: Attach tests to the right artifacts

A core reason golden sets drift is that they’re built around vague questions instead of concrete artifacts.

Make each test case explicit about its dependencies:

Doc version(s): Which policy or knowledge base version is assumed?
Prompt / agent version: Which agent config or prompt template it was written for?
Model: The target model family or tier (e.g., gpt‑4.x vs. internal SLM).

This is where versioning matters:

Store your test set in a system that automatically versions dataset collections.
Use Git (or similar) for prompts and evaluation configs.
Tie test runs to specific versions of:
- RAG index / embeddings
- Chunking strategy
- Query rewriting / retrieval pipeline

In Galileo’s workflow, your golden set lives as a versioned dataset, and every eval run is associated with:

Dataset version
Evaluator versions (including Luna‑2 updates)
Prompt and model versions

That gives you reproducibility: if a regression appears, you can replay the exact configuration that used to pass.

Step 4: Measure end-to-end task completion, not just point metrics

RAG reliability is more than per-question accuracy. Golden sets should also capture end-to-end tasks:

Multi-turn troubleshooting with several tool calls
Escalation flows (“If policy is unclear, suggest contacting Legal”)
Retrieval + summarization + action (e.g., generate an email or contract clause from docs)

For these, your tests should capture:

The sequence of steps / tools the agent should use (“golden path”)
Acceptable variation (“can call Tool A or Tool B; both are valid”)
Hard constraints (never execute this action without approval, never call this tool in EU data flows, etc.)

When you run evals, you’re not just asking “Is this answer grounded?” You’re asking:

Did the agent hit the right tools in the right order?
Did it avoid unsafe or unsupported actions?
Did it complete the user’s task in a minimal number of steps?

Galileo’s traces (sessions → traces → spans) and Evaluation Engine help you encode these as evaluators that track behavior across the full session, not just the final message.

Step 5: Keep your golden set in sync with docs and policies

This is the part most teams get wrong. A golden set that doesn’t evolve is just a snapshot.

1. Detect changes in your source of truth

Where do your policies live?

Confluence / Notion / Google Docs
Internal policy repositories or wikis
Git-backed policy-as-code repos

You need a pipeline (scripted or platform-based) that detects:

New docs added
Existing docs updated (especially high-risk ones)
Deprecated docs that should no longer be cited

When a critical doc changes, trigger:

Review of affected test cases: anything that references that doc or policy area
Creation of new tests: for new clauses, exceptions, or workflows introduced

A practical tactic:

Tag test cases with doc IDs or URLs.
Maintain a mapping from doc → tests.
On doc change, use that mapping to produce a “tests to review” queue.

2. Use production traffic as your change detector

Your users will always find paths you didn’t anticipate. Instead of waiting for tickets:

Run always-on evaluation on 100% of production traces.
Use evaluators like groundedness, safety, policy adherence, and “out-of-distribution” intent detection.
Let the system surface signals: clusters of failures, new query patterns, or emerging policy-confusion areas.

In Galileo, this is exactly what Signals does:

It continuously analyzes traces and spans.
It surfaces unknown unknowns (new question patterns, drift, cascading failures).
You can promote these surfaced examples into your golden test set with a few clicks.
You can even auto-generate a new LLM judge from a signal—turning a discovered pattern into a reusable evaluator.

This is how your golden set becomes a living asset: production itself tells you what to add.

3. Establish a review cadence

Policy and docs change on different cadences. Match your golden set updates to those:

Weekly: Review newly surfaced production patterns and candidate tests from Signals.
On every major doc/policy release: Run a focused review of all tests tagged with that policy area.
Before each significant release (prompt/index/model): Run your golden suite and block if key SLAs regress (e.g., groundedness below 99% on compliance queries).

Treat golden-set reviews like code reviews: required check before merge.

Step 6: Turn your golden set into production guardrails

If your golden set only runs offline, you still have a gap: users will see failures between eval runs.

The solution is to compress the logic of your golden set into real-time guardrails:

1. Distill evaluators into production-grade models

Heavy LLM-as-judge evaluators are precise, but too slow and expensive to run on every request.

Galileo’s approach:

Start with rich evaluators built from your golden set (including SME annotations and CLHF-tuned examples).
Distill them into compact Luna / Luna‑2 models optimized for evaluation tasks.
Serve them on a purpose-built inference stack with sub‑200ms latency.

Result: you can run your best evaluators continuously on 100% of traffic at up to 97% lower cost than heavyweight judges.

2. Attach explicit actions to evaluation outcomes

Don’t just log violations—intercept them:

With Protect, you can:

Score every input/output on:
- Groundedness / hallucination risk
- Policy adherence
- PII / secret leakage
- Tool-selection safety
Define guardrail policies that trigger:
- Block: Don’t show the answer; instead, respond with a safe fallback.
- Redact: Strip PII or sensitive terms before sending.
- Override: Replace the model’s output with a policy-approved template or escalation step.
- Webhook: Notify Slack, PagerDuty, or internal approval systems for high-risk cases.

Those policies are versioned, so you can iterate, test, deploy, and roll back without changing application code.

3. Close the loop: protect → evaluate → update

Whenever a guardrail fires:

Log the trace and span as a candidate test case.
Have SMEs review and, if appropriate, add it to your golden set.
Use it as a new few-shot example for your evaluators (via CLHF) to refine borderline behavior.

Over time, your golden test set, evaluation models, and guardrails all evolve together.

Practical patterns and anti-patterns

Patterns that work

Dataset collections with auto-versioning: Every change to your golden set is tracked, reproducible, and tied to evaluation results.
Mustache-based prompt templating: Create variations of test prompts (e.g., different regions, dates, products) from a single template to expand coverage without manual copy-paste.
Golden paths for end-to-end tasks: Explicitly define expected tool sequences for complex workflows and test against them.
Policy tags on tests: So doc updates automatically surface relevant tests for review.
Signals-driven expansion: New production behavior → signal → test case → evaluator → guardrail.

Anti-patterns to avoid

One-time “benchmark day”: Running a big eval once during launch, then never revisiting it while docs change weekly.
Generic, model-vendor evals: Using stock “helpfulness” scores that don’t know your policies or regulations.
Chat-with-logs as your only debugging tool: Powerful for forensics, but reactive and biased toward failures you know to search for.
Feature-flag-only guardrails: Toggling endpoints without real evaluation logic leaves you guessing whether you’re actually safer.

If you can’t run your best evaluators continuously in production, you don’t have reliability—you have a demo.

How Galileo helps you maintain a high-fidelity golden test set

To make this concrete, here’s how you’d operationalize all of this on Galileo:

Build your golden set
- Import real interactions and synthetic edge cases into dataset collections.
- Use mustache prompt templates and nested fields to generate variants.
- Have SMEs label expected answers and policy constraints.
Design evaluators from your golden set
- Use the Evaluation Engine with 20+ RAG and safety evaluators.
- Create custom evaluators from natural-language specs (e.g., “The answer must cite current EU data retention policy and must not mention deprecated 2019 rules”).
- Refine evaluators using CLHF with SME feedback and real traffic examples.
Run evals on every change
- Evaluate across dataset collections whenever you update docs, prompts, indexes, or models.
- Compare runs by version; block releases that regress on critical flows.
Watch production for drift
- Enable Signals on production traces to detect new failure modes and emerging query patterns.
- Promote surfaced examples into your golden dataset with tags and SME review.
Enforce guardrails in real time
- Distill evaluators into Luna‑2 and run them as part of Protect with sub‑200ms latency.
- Define guardrail policies to block, redact, override, or webhook on violations.
- Version guardrail configurations and roll back safely when needed.

This closes the eval-to-guardrail loop: your golden test set doesn’t sit in a binder—it drives how your RAG app behaves in production.

Summary

The best way to build and maintain a golden test set for a RAG app—especially when docs and policies change frequently—is to treat it as a living, versioned asset that:

Starts from real, high-risk workflows and SME-defined criteria.
Combines real interactions with synthetic edge cases and adversarial prompts.
Is tightly versioned with your docs, prompts, and models.
Continuously absorbs new patterns from production via Signals.
Powers real-time guardrails via distilled evaluators like Luna‑2, enforced by Protect.

That’s how you move from “we hope our RAG app is accurate” to “we can prove it—and we can stop bad answers before users see them.”

Next Step

Get Started

Answers you can trust, from Codeables