How do I create an eval dataset in Arize from production traces and run an offline experiment against a new prompt?

Most teams hit the same wall: you’ve got rich production traces from your agents, but you’re still guessing whether a new prompt will actually improve behavior or just move the failure modes around. The fix is to turn those real-world traces into an eval dataset, then run offline experiments that compare your current prompt to a new one before you ship.

Quick Answer: In Arize, you create an eval dataset by filtering and selecting production traces, saving them as a dataset, and attaching evaluators. Then you run an offline experiment that replays those test cases against a new prompt (or model) and compares eval scores side‑by‑side with your current baseline—no traffic shifting required.

Why This Matters

If you’re shipping prompts and agents without eval datasets grounded in production behavior, you’re essentially A/B testing on your users. Building eval datasets from real traces lets you capture the messy edge cases—tool failures, long sessions, ambiguous user intents—and then safely test new prompts offline. When you pair that dataset with structured evaluations, you get a repeatable loop: trace → curate → evaluate → deploy with confidence.

Key Benefits:

• Grounded in reality: Datasets built from production traces reflect actual user language, tools, and edge cases—not synthetic toy examples.
• Safe iteration: Offline experiments let you test new prompts, models, or retrieval strategies without running them live or risking SLO violations.
• Faster regression detection: Evaluator scores and span-level traces make it obvious when a new prompt helps one slice but hurts another.

Core Concepts & Key Points

Concept	Definition	Why it's important
Production trace	A full, OTEL-compatible record of an agent run in Arize (spans, tool calls, LLM calls, sessions, multi-agent graphs).	Gives you “the full flow” to sample real interactions, including failures and complex paths, for your eval datasets.
Eval dataset	A curated collection of production traces or trace-derived rows, stored in Arize as a reusable test set with labels/metadata.	Becomes your canonical “test suite” for prompts, models, and routing changes across iterations.
Offline experiment	A batch run in Arize that replays your eval dataset against one or more candidate prompts/models and runs evaluators on the outputs.	Lets you compare baselines vs. candidates side-by-side and gate releases using metrics instead of gut feel.

How It Works (Step-by-Step)

At a high level, you’ll:

1. Trace production behavior in Arize.
2. Curate an eval dataset from those traces.
3. Attach evaluators and run an offline experiment against a new prompt.

Below is a more detailed breakdown.

1. Make sure production is fully traced

One platform. Open standard tracing.

Before you can build a dataset, you need high-fidelity traces flowing into Arize (AX or Phoenix).

• Instrument with OTEL + OpenInference:

  • Wrap LLM calls, tool calls, and routers so each step emits spans.
  • Add attributes like user_intent, tool_name, latency_ms, status, and any business IDs you care about.
  • For multi-step agents, use parent/child relationships so Arize can render a multi-agent graph.

• Send traces to Arize:

  • Use Arize’s OTEL endpoint or the Phoenix SDK to export spans.
  • Confirm traces show up in the UI with:
    • The full span tree (LLM → tool → LLM…)
    • Request/response payloads (or redacted forms, if regulated)
    • Custom attributes and tags for later filtering.

You can’t create a good eval dataset from production if your traces only log “start” and “finish.” Make sure every decision point is a span.

2. Filter production traces to find good candidates

Next, you’ll carve out a subset of production behavior that reflects the tasks you care about and the failure modes you want to fix.

In Arize:

1. Open the Traces view:

   • Navigate to the service or agent you want to evaluate.
   • Use time range controls to target a representative window (e.g., last 7–30 days).

2. Apply filters to hone in on target scenarios:

   • By task type or intent: e.g., user_intent = "question_answering" or flow = "refund_policy"
   • By status or quality signals: e.g., status = "error", http_status = 5xx, or spans marked with low internal scores.
   • By tool usage: e.g., traces where a specific tool was called, or where tool calls frequently failed.
   • By user segment: e.g., region = "EU", plan = "enterprise" if you want SLO-critical slices.

3. Sample across difficulty and slices:

   • Include:
     • “Happy path” traces where things worked.
     • Known failure patterns (wrong tool, hallucinations, missing information).
     • Edge slices (long prompts, out-of-domain queries).
   • Aim for diversity over volume; 200–1,000 cases with good coverage often beats 10,000 near-duplicates.

This filtered set is your raw material for an eval dataset.

3. Save the filtered traces as an eval dataset

Once you’re happy with your filter:

1. Create a dataset from the selection:

   • In the Traces view, use the “Create Dataset” or “Save as Dataset” action.
   • Choose whether to:
     • Store entire traces (best for agent/e2e evaluations), or
     • Derive rows from specific spans (best for single LLM call tasks).

2. Define dataset fields:

   • Mark:
     • Input fields: user query, context, tool inputs.
     • Output fields: model response, tool outputs, final answer.
     • Metadata fields: task type, slice tags (e.g., jurisdiction, language), timestamps.
   • If you have ground truth labels in production (e.g., user ratings, internal correctness flags), map them into the dataset as label columns.

3. Name and version the dataset:

   • Use a clear naming scheme, for example:
     qa_refunds_prod_2025_04_eval_v1
   • This becomes your “test suite” reference in future experiments.

From here on, you should treat this dataset like code: versioned, reviewed, and reused across prompt and model changes.

4. Attach evaluators to your dataset

Evaluations turn a dataset from “a pile of traces” into a measurable benchmark.

In Arize you can use:

• LLM-as-a-Judge templates:

  • Tool selection correctness (did the agent call the right tools?).
  • Factuality / hallucination checks.
  • Instruction adherence, tone, and safety.
  • Path convergence (did different paths reach consistent answers?).

• Code-based evals:

  • Deterministic checks on structured outputs (JSON schema validity, required fields).
  • Business-rule assertions (e.g., total must equal sum of line items).

• Human annotation and queues (for critical slices):

  • Send especially important or ambiguous rows to a queue.
  • Have reviewers assign labels like “correct / incorrect” or detailed issue codes.
  • Use these as gold labels to calibrate LLM judges and as high-value test cases.

Practically:

1. Open your dataset in Arize.
2. Add evaluators:
   • Select from built-in templates in the Arize LLM evaluation framework.
   • Or bring your own evaluator functions, pointing to the appropriate input/output fields.
3. Run a dry-run eval:
   • Execute evaluators on a small subset to confirm they behave as expected and the prompts/criteria are correct.

Now your dataset isn’t just a log dump—it’s an instrumented benchmark.

5. Configure an offline experiment with a new prompt

Now you’re ready to test the new prompt against your existing baseline.

1. Define your baselines and candidates:

   • Baseline: your current production prompt (and model parameters).
   • Candidate: your new prompt version (or new model / routing configuration).

2. Create an experiment in Arize:

   • Go to the Experiments section.
   • Click “New Experiment” and:
     • Select your eval dataset.
     • Choose the baseline config (existing prompt).
     • Add one or more candidate configs (new prompts/models).

3. Set experiment scope and settings:

   • Decide whether to:
     • Reuse cached outputs for the baseline (if already logged), or
     • Re-run both baseline and candidate to remove temporal effects.
   • Configure sampling if you don’t need to run on the full dataset (e.g., 500 representative rows).

4. Attach the evaluators:

   • Select the evaluators you configured on the dataset (LLM-as-a-Judge, code evals, etc.).
   • Confirm each evaluator is mapped to the correct fields for both baseline and candidate outputs.

5. Run the experiment offline:

   • Launch the experiment.
   • Arize will:
     • Replay the eval dataset through each configuration.
     • Capture new traces for candidate runs.
     • Run all evaluators across baseline and candidate outputs.
     • Aggregate results into experiment metrics and dashboards.

No user traffic is touched; everything happens offline against your historical data.

6. Analyze experiment results and decide whether to ship

Once the run finishes, you’ll see side-by-side metrics across all evaluators.

Look for:

• Overall score deltas:

  • Did factuality scores go up or down?
  • Did tool-selection accuracy improve?
  • Did JSON validity or constraint checks change?

• Slice-level behavior:

  • Break down results by key slices: language, region, product type, user segment.
  • Ensure improvements aren’t masking regressions in high-risk slices.

• Trace-level debugging:

  • Drill into worst-performing candidate traces.
  • Use the span graph to see:
    • Whether the new prompt changed tool call patterns.
    • Where hallucinations or failures emerged.
    • Latency and token/cost changes.

Once you’re confident:

• If candidate ≥ baseline on critical evaluators and slices:
  • Promote the new prompt to production (ideally via CI/CD Experiments that gate releases on these metrics).
• If candidate regresses important metrics:
  • Keep iterating on the prompt.
  • Use failing traces as new additions to your eval dataset (especially via annotation queues).

This is the build–learn–improve loop: production traces feed datasets, datasets feed experiments, experiments gate production.

Common Mistakes to Avoid

• Using synthetic or cherry-picked examples only:
  Relying solely on handcrafted test cases misses real-world complexity. Always seed your eval dataset from production traces and then augment as needed.

• Evaluating only aggregate accuracy:
  A single average score can hide critical regressions in specific slices. Break down experiment results by task type, region, user segment, and error mode before you ship.

Real-World Example

At my current marketplace, our support agent regularly hallucinated refund policies for certain countries. We instrumented every agent run with OpenTelemetry, then used Arize to filter traces where the intent = "refund_policy" and the region IN ("DE", "FR", "ES"). We saved ~600 of these traces as a dataset, added an LLM-as-a-Judge eval for “policy correctness vs. internal spec,” and sent 100 hard cases through a human annotation queue for gold labels.

We then created an offline experiment in Arize comparing our existing prompt to a new, policy-grounded prompt that forced retrieval from an internal tool. The results: overall correctness improved from 71% to 90%, with zero regressions on safety criteria and only a small latency increase. Because everything was based on production traces—and we could inspect spans where the candidate still failed—we were comfortable gating our CI pipeline on these metrics and rolling out the new prompt for EU customers.

Pro Tip: Each time an experiment reveals a new failure mode (e.g., a tricky phrasing or a tool timeout pattern), add that trace to your eval dataset and, if needed, your annotation queue. Over time, your dataset becomes a living, self-improving test suite that mirrors your real production risk surface.

Summary

Creating an eval dataset in Arize from production traces—and using it to run offline experiments against new prompts—is the most reliable way to ship agents that work. Instrument your system with open standard tracing, curate a diverse slice of real production behavior, attach evaluators that reflect your actual quality bar, and then compare baselines and candidates side-by-side before any change hits live traffic. When you close this loop and treat datasets and experiments like part of your CI/CD, you move from impressive demos to reliable production systems.

Next Step

Get Started