How do I run CI/CD experiments in Arize to gate prompt/model changes before merging?

Quick Answer: In Arize, you gate prompt/model changes by turning each change into a versioned experiment, running offline and online evals on real datasets, and wiring those eval results into CI/CD so only variants that clear your quality thresholds are allowed to merge. You trace every request, score every span or output, and let experiments act as the approval check on your PR or deployment pipeline.

Why This Matters

In non-deterministic systems like LLMs and agents, “it worked in staging” is meaningless without evidence across real workloads. CI/CD experiments in Arize give you that evidence: every prompt, model, or retrieval tweak is run as a structured experiment against curated datasets and production traces, scored with LLM and code-based evaluations, and compared side‑by‑side with the current baseline. The result is a build–measure–ship loop where regressions are caught before they ever hit your SLOs or customers.

Key Benefits:

• Objective gating for prompt/model changes: Replace subjective “looks good” checks with eval-driven thresholds that automatically block risky merges.
• Tighter feedback loop with production: Use real spans and sessions from production as your experiment input so you’re testing on the edge cases that actually hurt you.
• Traceable, repeatable releases: Every experiment run is stored with prompts, models, metrics, and decisions, giving you a clean audit trail and easy rollback paths.

Core Concepts & Key Points

Concept	Definition	Why it's important
Experiments	Structured runs that compare prompt/model/agent variants on the same dataset with shared evaluation criteria.	Let you quantify impact (accuracy, hallucination rate, tool-call correctness) before you change production.
Offline & Online Evals	Offline evals score historical or curated data; online evals continuously score live production traffic.	Offline evals power pre-merge CI/CD checks; online evals ensure that behavior doesn’t drift after deployment.
CI/CD Gating	Using experiment metrics and thresholds as automated checks in your deployment pipeline.	Prevents prompt or model regressions from merging, similar to unit tests for deterministic services.

How It Works (Step-by-Step)

At a high level, you’ll: instrument traces, curate datasets, define evals, create experiments for each change, and wire the experiment results into your CI/CD system as a gate.

1. Instrument and Trace Your Agents with OpenTelemetry

One platform. Open standard tracing.

To run meaningful CI/CD experiments, you need rich traces that capture the full flow for each request:

1. Standardize on OTEL and OpenInference:

   • Instrument your LLM/agent stack with OpenTelemetry and OpenInference-style span schemas.
   • Log prompt templates, model names, tool calls, tool inputs/outputs, retrieval steps, and final responses as spans within a trace.
   • Prefer explicit attributes for key fields (e.g., llm.prompt_template, llm.model_name, tool.name, tool.success) so they’re queryable.

2. Send traces to Arize (AX or Phoenix):

   • Forward spans from your tracing pipeline (e.g., OTEL Collector) into Arize.
   • Ensure you’re logging stable IDs like session_id, trace_id, and a business key (e.g., conversation_id or order_id) to correlate sessions and metrics.

3. Verify visibility:

   • In Arize, inspect a few traces and multi-agent graphs.
   • Confirm you can see every agent step, tool call, and LLM response — if you can’t trace it, you can’t evaluate it in CI.

2. Curate Datasets from Production Traces

Experiments are only as good as the datasets behind them.

1. Select slices that matter:

   • Use Arize’s search and filters to pull:
     • High-value flows (checkout, KYC, refunds, etc.).
     • Known failure modes (hallucinations, routing errors, tool failures).
     • Edge cases (long contexts, multilingual, high token counts).
   • Save these as datasets in Arize so they can be reused across experiments.

2. Create golden subsets with annotation queues:

   • Use Human Annotation and Queues to:
     • Label correct answers and acceptable tool paths.
     • Mark hallucinations, unsafe content, and poor tool usage.
   • These become your “golden datasets” for regression testing.

3. Version your datasets:

   • When you change criteria or add new cases, create a new dataset version.
   • Reference dataset versions explicitly in CI so you know what each experiment tested against.

3. Define Evaluations (LLM, Code, and Human)

Evaluation that powers reliable, production-ready agents.

Arize supports both offline and online evals; you’ll use offline evals as your primary CI gate and online evals to watch production after merge.

1. Add LLM-as-a-Judge evals:

   • Use Arize’s eval templates or your own prompts to score:
     • Task correctness (Did the answer match the ground truth?)
     • Hallucination risk (Is the answer unsupported by retrieved context?)
     • Tool selection & path quality (Were the right tools used? Did the agent converge efficiently?)
   • Configure them as reusable evaluators bound to datasets or spans.

2. Add code-based evals for deterministic checks:

   • For tasks with programmatic truth (e.g., SQL correctness, JSON schema, path equivalence), use code evals:
     • Parse the output and compare to expected values.
     • Validate structured responses (e.g., all required keys present, types correct).
   • These act like unit tests but applied to LLM/agent outputs.

3. Wire in human review where needed:

   • For ambiguous cases (tone, UX quality, borderline safety), funnel spans into annotation queues.
   • Use these annotations both as labels and as ground truth for future LLM-as-a-Judge prompts.

4. Create an Experiment for Each Prompt/Model Change

CI/CD experiments.

Every meaningful change — new prompt template, model version, router configuration, or tool strategy — should correspond to an experiment.

1. Define the baseline and candidate(s):

   • Baseline: your current production prompt/model/config.
   • Candidate: the new version from your branch (e.g., updated system prompt, different model, new retrieval k).
   • In Arize, define these as variants, each with:
     • Prompt or rule file version.
     • Model and parameters.
     • Any agent/router flags (e.g., “tool-first,” “reflection enabled”).

2. Attach datasets and evaluators:

   • Select the datasets you curated in Step 2.
   • Attach your LLM-as-a-Judge and code evaluators:
     • Correctness, hallucination, safety, tool correctness, latency, cost, etc.
   • Configure metrics (e.g., accuracy, pass rate, mean score) to compute for each variant.

3. Run the experiment:

   • Arize replays each dataset example against both baseline and candidate variants.
   • For each span/trace, Arize:
     • Executes the variant.
     • Logs new spans.
     • Applies all configured evals.
   • All runs are stored as a structured experiment with side-by-side metrics and visualizations.

4. Define pass/fail criteria:

   • For example:
     • “Candidate must have ≥ baseline accuracy and ≤ baseline hallucination rate.”
     • “No more than +10% latency or +5% cost per request.”
     • “Tool-call correctness must not degrade on any protected slice (e.g., KYC, refunds).”
   • These thresholds are what your CI pipeline will enforce.

5. Integrate Experiments with Your CI/CD Pipeline

Close the loop between AI development and production.

Now you tie experiment results into your PR and deployment flow so merges are gated by evaluations, not vibes.

1. Automate experiment runs from CI:

   • In your CI (GitHub Actions, GitLab CI, Jenkins, etc.):
     • When a PR changes prompts, models, or agent logic, trigger:
       • A call to Arize’s experiment run API (or CLI wrapper).
       • Passing the dataset IDs, baseline and candidate config, and target metrics.
   • Optionally, use MCP or an internal agent (like Alyx) to generate experiment configs from diffs.

2. Wait for experiment completion:

   • Poll Arize or subscribe to webhooks for experiment status.
   • Once complete, fetch:
     • Metric summaries.
     • Variant comparisons.
     • Whether thresholds were met.

3. Enforce gating logic:

   • Convert your pass/fail criteria into CI conditions, for example:
     • Block PR if:
       • accuracy(candidate) < accuracy(baseline)
       • OR hallucination_rate(candidate) > hallucination_rate(baseline) + 1%
       • OR any critical slice metric regresses.
   • Fail the job and annotate the PR with a summary:
     • “Prompt v3.4 rejected: hallucination +3.2% vs baseline on refunds dataset.”

4. Promote only passing variants:

   • If the experiment passes:
     • Tag the prompt/model version as “approved for production.”
     • Update your prompt hub, router, or config store to point production to the new version on merge.
   • Store the experiment ID in your deployment metadata for future audits.

6. Monitor with Online Evals After Merge

Evaluate production data.

Even with strong offline evals, LLM and agent behavior can drift under real traffic.

1. Turn on online evals:

   • Configure Arize to run Online Evals on live production spans:
     • Use the same LLM-as-a-Judge and code evals you used offline.
     • Focus especially on high-risk flows and known edge cases.

2. Set up dashboards and alerts:

   • Build dashboards that track:
     • Accuracy, hallucination rate, safety incidents.
     • Tool-call correctness and latency.
     • Cost per request.
   • Add alerts for:
     • Sudden drops in quality.
     • Spikes in failed tool calls.
     • Violations on regulated flows (e.g., KYC).

3. Feed incidents back into datasets:

   • When online evals or alerts surface new edge cases:
     • Add those spans to annotation queues.
     • Label them and promote them into your offline datasets.
   • This keeps your CI/CD experiments aligned with real-world behavior.

Common Mistakes to Avoid

• Treating evals as one-off checks instead of reusable assets:
  Build evaluators as reusable templates tied to datasets and slices, so every future experiment benefits from the same standards.

• Only testing the “happy path” flows:
  Make sure your datasets include high-risk slices, known failure modes, and rare edge cases; otherwise your gating logic can still let regressions through.

• Ignoring cost and latency in gating criteria:
  Evaluate not just correctness but also tokens, cost, and latency so you don’t accidentally ship a “better” prompt that breaks your rate limits or SLOs.

• Not versioning prompts and datasets:
  Always version prompts, models, and datasets; without this, it’s impossible to reproduce experiment results or understand which change actually improved things.

Real-World Example

At my current marketplace, we rebuilt our “buyer support agent” from a single LLM call into a multi-agent system orchestrating several tools (orders, refunds, policy, fraud). Early on, we shipped a prompt tweak that quietly increased hallucination on refund eligibility — it still “looked good” in staging, but production traces showed subtle policy violations.

After that incident, we moved all agent changes behind Arize CI/CD experiments:

• We instrumented every tool call and agent hop with OTEL and OpenInference conventions, then streamed spans into Arize.
• We curated datasets from refund, fraud, and KYC flows, and asked our operations team to annotate tricky cases via labeling queues.
• We defined LLM-as-a-Judge evals for “policy adherence,” “hallucination,” and “tool correctness,” plus code evals checking that refund outputs matched policy tables.
• For every PR touching prompts or routing, CI now launches an experiment comparing the branch’s variant to the current baseline across those datasets.
• If policy adherence drops even 0.5% on refunds or KYC slices, the experiment fails and the PR can’t merge.

Within a few weeks, engineers stopped arguing about “is this change better?” and started shipping smaller, safer iterations. We caught multiple regressions on tool selection (e.g., using the generic “order lookup” tool instead of a policy-specific one) before they ever reached production.

Pro Tip: Start by gating a narrow, high-risk slice (like refunds or KYC) instead of every agent change. Once you’ve proven the CI/CD experiment pattern there, expand your datasets and evals to cover more flows without slowing the whole team down.

Summary

Running CI/CD experiments in Arize to gate prompt/model changes means treating every change like a hypothesis that must prove itself on real data before it ships. You trace the full agent flow with OTEL, curate datasets from production, define LLM and code evals, and run structured experiments that compare your new variant to a baseline. Those experiment results become a hard gate in your CI/CD pipeline, while online evals watch production and feed new edge cases back into your datasets. This is how you move from demoes to “Ship Agents that Work” in environments with real SLOs and compliance constraints.

Next Step

Get Started