How do I set up a human annotation queue in Arize so reviewers can label bad outputs and we can build a golden dataset?

Most teams add humans to the loop only after a bad incident. The better pattern is to stand up a human annotation queue early, route suspicious spans into it, and steadily turn those edge cases into a golden dataset that hardens your agents over time.

Quick Answer: In Arize, you set up a human annotation queue by (1) deciding what “bad output” means for your use case, (2) configuring annotations and tasks in an annotation queue, and (3) piping production spans into that queue based on filters (e.g., low eval scores, specific tools, or slices). Reviewers then label spans in one place, and Arize turns those annotations into a reusable golden dataset you can feed back into evals, experiments, and prompt optimization.

Why This Matters

If you’re shipping agents into production, you will see hallucinations, tool misuse, and subtle policy violations that automated evals miss. A structured human annotation queue gives you a single place to triage these bad outputs, score them consistently, and promote the most informative cases into a golden dataset. That dataset is what powers tighter evaluations, more realistic experiments, and CI/CD checks that prevent regressions from slipping into production.

Key Benefits:

• Centralized review: Reviewers see the full trace for each bad output—prompts, tool calls, and responses—in one queue instead of scattered screenshots or tickets.
• Consistent labels: Shared schemas and annotation templates make reviewers grade spans the same way, so your evals and experiments rely on clean, comparable labels.
• Golden datasets that improve over time: The worst and weirdest production cases are captured, labeled, and reused to harden prompts, models, and agents instead of getting lost in logs or chat threads.

Core Concepts & Key Points

Concept	Definition	Why it's important
Annotation	A structured label or score attached to a span or response (e.g., is_bad_output=true, clarity=unclear, policy_violation=PII)	Turns qualitative human judgment into queryable, repeatable signals you can slice, aggregate, and optimize against.
Annotation Queue	A workflow in Arize where spans are batched into review tasks for human annotators to label according to a defined schema	Centralizes review, streamlines labeling, and ensures every reviewer looks at the right spans in a consistent UI.
Golden Dataset	A curated collection of annotated spans that represent real production behavior and edge cases	Becomes the backbone for LLM-as-a-judge prompts, regression tests, experiments, and CI/CD gates.

How It Works (Step-by-Step)

One platform. Arize connects open-standard tracing with annotation queues so you can move from “we saw a bad output” to “this edge case is now part of our regression suite.”

1. Decide what “bad output” means for your team

Before you open the UI, get crisp on what reviewers should look for. Common categories:

• Factual issues: hallucinations, incorrect numbers, missing key facts
• Tool issues: wrong tool selection, wrong parameters, tool not called when required
• Policy issues: PII leakage, disallowed content, unsafe instructions
• UX issues: unclear, incomplete, or low-utility responses

Write these as concrete annotation fields you want on each span, for example:

• overall_quality (1–5)
• is_bad_output (yes/no)
• error_type (enum: hallucination, tool_error, policy_violation, formatting, other)
• should_be_in_golden_dataset (yes/no)

These definitions are what your reviewers will use; they also become the schema for later eval prompts and code checks.

2. Instrument your agents so spans are traceable

Human annotation is only useful if reviewers can see “the full flow.” Use Arize’s Open Standard Tracing—powered by OTEL and OpenInference conventions—to log:

• Incoming user request (with a stable trace/session ID)
• System prompt and any routing or policy prompts
• Tool calls: tool name, arguments, responses, timing
• Intermediate agent messages and final answer
• Any existing auto-eval scores (LLM-as-a-Judge, code evals)

In practice this means:

• Adding OTEL spans for each tool call and model invocation
• Propagating trace and span IDs through your stack
• Logging structured attributes (e.g., user_segment, use_case, model_name, agent_version)

Once these spans are flowing into Arize, you’ll be able to search and filter for exactly the slices you want humans to review.

3. Configure an annotation schema and queue

Next, you define how reviewers should label each span and set up the queue that assigns work.

1. Define annotation fields
   In Arize, design fields that reflect the decisions you made earlier, for example:

   • overall_score (int 1–5)
   • is_bad_output (bool)
   • primary_issue (enum)
   • notes (free text)
   • include_in_golden_dataset (bool)

2. Create an annotation queue

   • Scope the queue: is it for one agent, one use case, or cross-product?
   • Select which spans are eligible (e.g., “agent_name=search_assistant AND env=prod”).
   • Attach the annotation schema so every task has the same fields.

3. Define reviewer roles and volume

   • Decide who will label: internal SMEs, support agents, or a small reliability team.
   • Set rough quotas: e.g., “50 spans/day from production low-score cases,” so you don’t overwhelm reviewers.

4. Route the right spans into the queue

You don’t want to label everything—just the most informative failure modes. Use Arize’s filtering to push spans into queues based on:

• Online eval scores:
  • Example: llm_eval:helpfulness_score < 3 or llm_eval:tool_selection_pass=false
• Errors and retries:
  • Spans with exceptions or repeated tool calls
• User behavior:
  • Sessions where users quickly re-asked, abandoned, or gave explicit negative feedback
• Critical slices:
  • Regulated markets, specific geos (for policy nuances), or high-value customers
• New changes:
  • Spans from the latest agent_version or prompt experiment that you want to watch closely

You can start with a simple rule, such as “all spans with an online eval score below 3 and all spans from the new agent version,” and refine over time.

5. Have reviewers label bad outputs in Arize

Now your annotation queue is live and populated. Reviewers work entirely inside Arize:

1. Open the queue and pull the next task.
2. Inspect the trace:
   • See system and user prompts
   • Check tool calls and responses
   • View the final answer and any auto-eval scores
3. Apply annotations using the schema:
   • Mark is_bad_output and primary_issue
   • Give an overall_score
   • Flag include_in_golden_dataset if the example is representative and useful for future tests
4. Submit the task; Arize attaches annotations directly to that span.

Because traces are logged via OTEL/OpenInference, reviewers see the complete multi-step, multi-agent path rather than a single response in isolation. That context is critical when you’re trying to distinguish “model hallucination” from “tool returned stale data” or “prompt routed incorrectly.”

6. Turn annotations into a golden dataset

Once you’ve labeled a few hundred spans, you can slice down to the gold:

1. Filter for high-signal examples

   • include_in_golden_dataset=true
   • Or is_bad_output=true AND env=prod AND agent_version in (latest, previous)

2. Export or promote to a dataset

   • Use Arize’s datasets to save this slice as “search_assistant_golden_bad_outputs_v1” (or multiple datasets per use case).
   • Each row now has: input context, tools, final output, eval scores, and human labels.

3. Connect this golden dataset back into your build loop

   • LLM-as-a-Judge templates: Use the human labels as ground truth to calibrate or train your evaluation prompts (e.g., for hallucination detection, policy compliance, or tool selection).
   • Experiments: When you test new prompts, retrieval strategies, or models, run them against this golden dataset in Arize Experiments and compare metrics like “bad_output_rate” or “policy_violation_rate.”
   • CI/CD gating: Wire the golden dataset into your CI/CD Experiments so any change that increases the bad output rate on these gold examples is blocked from production.
   • Prompt optimization: Use the worst failures in the dataset to replay traces in Arize’s playground, iterate on prompts or agent logic, and re-run evaluations.

Over time, your golden dataset becomes less about volume and more about coverage: the weird, multi-step, high-risk edge cases that define whether you’re ready to ship a new agent version.

Common Mistakes to Avoid

• Treating annotations as unstructured feedback:
  If reviewers just write free-text comments, you can’t query or automate against them. Instead, define explicit fields and enums and keep free text as a supplement.

• Labeling random spans instead of targeted slices:
  Random sampling gives you noise, not insight. Use filters (low eval scores, new agent versions, high-risk segments) so annotation time focuses on the cases that matter most to reliability and safety.

Real-World Example

At my last org, we had a marketplace search assistant that occasionally surfaced clearly wrong items for regulated categories. We instrumented the entire agent flow with OpenTelemetry, sent traces into Arize, and turned on Online Evals to score “relevance” and “policy compliance.” Any span with low relevance or a policy warning was automatically routed into a “regulated_search_review” annotation queue.

Support specialists reviewed ~50 spans per day. For each, they marked is_bad_output, issue_type (hallucination, policy, routing), and whether it should go into the golden dataset. Within a few weeks we had ~700 high-quality, annotated failures. We turned that into a golden dataset, wired it into Arize Experiments, and started gating prompt and routing changes in CI/CD. The result: we caught several regressions before they reached production and cut real-world bad outputs in those regulated categories by more than half, without guessing in the dark.

Pro Tip: Start with a narrow, high-impact queue (e.g., one agent + one risk slice) and a small, crisp schema. Once that’s working, expand to more use cases. It’s better to deeply annotate 300 critical spans than shallowly tag 30,000 random ones.

Summary

Setting up a human annotation queue in Arize is how you stop treating bad outputs as one-off incidents and start treating them as renewable assets. By defining clear labels, routing high-signal spans into a queue, and promoting the best examples into a golden dataset, you close the loop between production behavior and development. That golden dataset then powers better LLM-as-a-Judge prompts, more realistic experiments, and CI/CD checks that keep prompt and agent regressions out of production.

Next Step

Get Started