What’s a practical way to build a “golden dataset” for an agent and keep it updated as the product evolves?

Most teams don’t fail at agent reliability because they picked the “wrong” model—they fail because they can’t agree on what “good” looks like, or they can’t keep that definition in sync with a fast-moving product. A golden dataset is the antidote: a living, versioned set of real queries, edge cases, and labels that your agent must pass before you ship anything new.

Quick Answer: Build a golden dataset by starting small with high-impact, real user queries, labeling every step of the agent flow (not just final answers), and wiring that dataset directly into your evals and CI/CD. Keep it updated by turning production failures and edge cases into new test cases via annotation queues, then re-running experiments and gating releases on eval scores.

Why This Matters

If you don’t define “good” in data, you end up shipping agent changes on vibes: a handful of spot checks in a playground, a few ad hoc prompts in Slack, then a push to production. That works for demos, not for a marketplace with SLOs, audits, and real money at stake.

A practical golden dataset gives you:

• A common contract across product, eng, and compliance: “These are the scenarios our agent must handle.”
• A safety net for iteration: every new prompt, model, or tool configuration has to beat (or at least match) current performance on that dataset.
• A feedback loop from production: real failures and weird multi-step paths become future test cases, not recurring incidents.

Key Benefits:

• Catch regressions early: Run evals on your golden dataset in CI/CD so prompt and agent changes fail fast instead of failing in front of users.
• Align quality across teams: Use labeled examples as the shared source of truth for “correctness,” “safe,” “on-brand,” and “compliant.”
• Continuously improve the agent: Turn production traces and incidents into new golden test cases so the agent steadily hardens against edge cases.

Core Concepts & Key Points

Concept	Definition	Why it's important
Golden dataset	A curated, versioned set of representative queries, contexts, and labeled expected behaviors for your agent, covering both happy paths and failure modes.	It becomes your contract for correctness and the benchmark for comparing prompts, models, and routing strategies.
Step-level evaluations	Evaluations on each stage of the agent (router, retrieval, tool calls, reasoning, final answer) instead of only the final output.	Lets you pinpoint where things went wrong—hallucination vs tool misuse vs bad routing—and improve specific components.
Production-driven updates	A process that continuously mines production traces, incidents, and user feedback for new examples and adds them to the golden dataset.	Keeps your dataset relevant as the product, tools, and user behavior evolve, avoiding “stale benchmark” syndrome.

How It Works (Step-by-Step)

One platform. Build the dataset, wire evals into your loop, and keep everything updated with production data.

Below is the practical flow I use in production environments, mapped to how we run it with Arize AX and Phoenix, but the principles are vendor-agnostic and aligned with open standards (OpenTelemetry, OpenInference).

1. Define the surfaces and failure modes that matter

Before you collect any data, define the scope. For a typical agentified marketplace, I start with:

• Surfaces: support assistant, seller tools copilot, internal CX agent.
• Tasks: refund policy Q&A, dispute classification, listing policy checks, identity verification flows.
• Failure modes to cover:
  • Hallucinated policies or prices
  • Wrong tool or no tool when one is required
  • Wrong parameters to tools (amount, currency, user IDs)
  • Unsafe / non-compliant responses (legal, brand, region)
  • “Looks right but is wrong” edge cases (policy exceptions, complex routes)

Write this down. Your golden dataset is not “some questions”; it’s a set of essential behaviors and failure modes you care about.

2. Seed the first version of your golden dataset

Start small but intentional. Aim for 50–200 examples, not thousands.

Where to get seed examples:

• Historical tickets and chats: Mine real user conversations for:
  • Frequent intents (top 10 reasons for contact)
  • Known tricky edge cases (multi-policy overlaps, international rules)
• Incident postmortems: Every past outage or escalation is a test case waiting to happen:
  • If an agent previously approved an invalid refund, that scenario belongs in your golden dataset.
• Product / policy documents: Turn key spec bullets into prompts:
  • “What’s the refund policy for digital goods purchased more than 14 days ago in the EU?”
• Synthetic variations (optional): Once you have a few canonical examples, you can generate variants to stress-test paraphrases and formats, but keep a human in the loop for review.

For each example, capture at minimum:

• User input: Raw text (and structured inputs if relevant).
• Context: Any relevant state (user role, geo, account flags) and documents that should be retrieved.
• Expected outcome:
  • Final answer (natural language) and/or
  • Tool calls (which tool, parameters) and/or
  • Classification labels (intent, policy code, risk level, etc.).

Store this in a standard format—JSONL or Parquet—aligned with OpenInference-style fields so it’s portable across tools and frameworks. No proprietary schema lock-in.

3. Label step-by-step behavior, not just the final answer

Agents are multi-step systems. A golden dataset that only labels the final answer won’t help you understand where things break.

For each example, add labels for these stages where applicable:

1. Router / planner:

   • Did it choose the right path? (e.g., “billing issue” → refund tool flow)
   • Did it need tools at all, or was a direct answer appropriate?

2. Retrieval / context selection:

   • Were the right documents retrieved?
   • Were critical documents missing, or irrelevant ones included?

3. Tool calls:

   • Did the agent call the correct tools?
   • Were parameters extracted correctly? (amount, IDs, timestamps)
   • Was the execution order correct?

4. Reasoning and aggregation:

   • For multi-step flows, did the agent properly combine tool outputs?
   • Did it correctly apply business rules and constraints?

5. Final answer:

   • Correctness vs ground truth or policy.
   • Safety/compliance (no promises we can’t legally make).
   • Tone/brand alignment if that matters for your product.

In Arize, these map cleanly onto spans and sub-spans:

• Router span
• Retriever span(s)
• Tool call spans (with parameters)
• LLM reasoning span(s)
• Final response span

Each span can be evaluated by a dedicated evaluator (LLM-as-a-Judge, code-based, or human labels). That’s how you turn abstract “golden data” into actionable, step-level scores.

4. Turn labels into evaluators

Once you have labeled examples, you want automatic, repeatable checks.

Types of evaluators I recommend:

1. Code evals (deterministic):

   • For anything that can be expressed as a function: correct tool choice, valid JSON, parameter ranges, presence/absence of specific policy codes.
   • Example: check_refund_amount(request, tool_call) -> PASS/FAIL.

2. LLM-as-a-Judge evals (structured prompts):

   • For semantic checks: answer correctness, policy compliance, reasoning quality, tool choice explanation.
   • Use strict templates that return structured labels, not free-form prose.
   • Example dims: correctness, policy_compliance, tool_selection_quality, instruction_following.

3. Human annotation:

   • For new domains, gray areas, and training your LLM judges.
   • Run via annotation queues with clear rubrics: what counts as “correct,” “acceptable workaround,” and “hard fail.”

In Arize AX/Phoenix, golden dataset examples feed directly into evaluators attached to spans and traces. The same evaluators run offline (experiments) and online (monitors).

5. Wire the golden dataset into CI/CD

This is the “practical” part most teams skip. A golden dataset sitting in a spreadsheet doesn’t help you ship agents that work.

For every change—new prompt, router policy, model, or retrieval strategy:

1. Create a candidate version:

   • E.g., new prompt template or adjusted tool-calling instructions.

2. Run an experiment on the golden dataset:

   • Replay all examples through both current and candidate versions.
   • Evaluate each with your evaluator suite (code + LLM judges + any existing human labels).

3. Gate on eval scores:

   • Define thresholds and guardrails:
     • No regression on safety/compliance.
     • No more than X% drop in any critical dimension.
     • Preferably, statistically significant improvement in at least one key metric.
   • In Arize, this is CI/CD Experiments: you compare variants, slice by scenario, and block rollout if a candidate underperforms.

4. Log outcomes for traceability:

   • Keep a history of:
     • Golden dataset versions
     • Prompt/model versions
     • Eval scores
     • Rollout decisions

This is what “Close the loop between AI development and production” actually looks like in practice.

6. Continuously mine production for new golden cases

Static golden datasets rot. Your product changes, user behavior shifts, policies update. The only sustainable approach is to feed production traces back into your dataset.

Here’s the cycle that’s worked best for me:

1. Instrument everything with open standards:

   • Use OpenTelemetry and OpenInference-style schemas to trace:
     • User request spans
     • Tool calls
     • Intermediate reasoning
     • Final responses
     • Cost and latency
   • Send traces to Phoenix/AX so each request produces a full multi-span trace.

2. Run online evals in production:

   • Attach the same evaluators (code + LLM-as-a-Judge) to production traces.
   • Compute scores per span: tool correctness, policy compliance, hallucination risk, etc.
   • Set up dashboards and alerts for:
     • Drops in compliance
     • Spikes in tool-call errors
     • Slices with low correctness (e.g., “EU sellers, digital goods”).

3. Create annotation queues for the worst cases:

   • Triage traces into annotation queues based on:
     • Low eval scores
     • High business impact (high-value accounts, sensitive actions)
     • New patterns (new product features, new tools)
   • Have annotators:
     • Label what should have happened (ground truth).
     • Tag failure modes (wrong tool, misrouted, hallucinated, missing context).

4. Promote annotated traces to the golden dataset:

   • Periodically (weekly/bi-weekly), review annotated traces.
   • Select:
     • New edge cases not currently covered.
     • Regenerated versions of existing cases where policy or product changed.
   • Add them as new golden examples, with:
     • Full context (input, state, docs, tools).
     • Expected behavior and step-level labels.
   • Version your golden dataset (v1, v2, …) so you can track the evolution.

5. Re-run experiments on new versions:

   • Whenever the golden dataset updates:
     • Recompute baseline scores for your current production agent.
     • Ensure future candidates are compared against the new baseline.

That’s how you keep the dataset in lockstep with reality instead of letting it drift into irrelevance.

Common Mistakes to Avoid

• Treating golden datasets as single-shot projects:
  Teams do a big labeling push once, then never update. Avoid this by baking golden dataset updates into your incident and postmortem process: every major incident should create at least one new test case.

• Only evaluating final answers:
  If you don’t evaluate routers, tools, and retrieval, every regression looks like “the model is bad.” Instrument spans and attach step-level evals so you can actually fix the right layer.

• Relying on one black-box eval model:
  One proprietary judge with unknown behavior is a hidden dependency. Use open-source LLM judges, code evals, and human labels together—and keep prompts and judge models versioned and inspectable.

• Skipping online evals:
  Offline-only testing misses real user behavior and long-tail edge cases. Use online evals in production to surface weird paths and then promote those traces back into your golden dataset.

Real-World Example

At my current company (a global marketplace), we rolled out a policy enforcement agent used by both buyers and sellers. Early on, every change was tested via manual spot checks—engineers pasting a dozen prompts into a playground and calling it “good enough.” Two things happened:

1. A prompt change accidentally loosened enforcement on a niche content policy for one region. It slipped through because none of our “playground prompts” covered that edge case.
2. We discovered during an audit that similar queries in different languages were handled inconsistently, despite sharing the same underlying policy.

We fixed this by building a practical golden dataset:

• Seeded from reality: We pulled 200 historical tickets covering the top 20 policy categories and all regions, plus 30 examples from past incidents and escalations.
• Labeled end-to-end: For each, we annotated:
  • Expected policy code(s)
  • Required tools (policy checker, translation)
  • Correct parameters (region, category, severity)
  • Final outcome (approve, reject, escalate) + justification.
• Step-level evals: We wrote:
  • Code evals for correct policy IDs and tool parameters.
  • LLM-as-a-Judge evals for “policy compliance” and “instruction following.”
• CI/CD Experiments: Every prompt or model change now runs against this golden dataset in Arize:
  • Changes are auto-blocked if policy compliance drops on any high-risk slice (e.g., minors, restricted goods).
• Production feedback loop: We:
  • Instrumented agents with OTEL tracing, so every request logs the router, tools, reasoning, and final answer.
  • Ran online evals to flag low-compliance traces.
  • Sent those traces into annotation queues.
  • Promoted new, high-impact examples into the golden dataset every two weeks.

Within a quarter:

• We caught a router bug in staging that would have misrouted 8% of EU policy queries.
• We hardened the agent against several weird multi-step paths our initial dataset never imagined.
• Auditors were much happier because we could show a versioned golden dataset and eval history, not just “we tested it manually.”

Pro Tip: Don’t wait until your golden dataset is “perfect” to wire it into CI/CD. Start with 30–50 high-impact cases, gate changes on those, and then grow the dataset using production traces and incidents. Perfection is less important than having a consistent bar you enforce on every change.

Summary

A golden dataset for an agent is not a static benchmark; it’s a living contract for how your system should behave across the paths and edge cases that matter most. The practical way to build and maintain it is:

• Start with real, high-impact queries and past incidents as seed examples.
• Label every step of the agent flow—router, retrieval, tools, reasoning, final answer—and turn those labels into code and LLM-as-a-Judge evals.
• Wire the dataset into experiments and CI/CD so every change is measured before it reaches production.
• Use open-standard tracing and online evals to mine production traces and incidents, then feed those back into annotation queues and, eventually, into the next version of your golden dataset.

Do that, and you stop demoing—and start shipping agents that actually work.

Next Step

Get Started