How can I capture real user prompts and model outputs to build an evaluation dataset from production traffic?

Most teams discover that their best evaluation data is already flowing through their production systems—they just don’t have a safe, structured way to capture it. Turning real user prompts and model outputs into an evaluation dataset is one of the fastest ways to improve model quality, reduce regressions, and systematically tune prompts or models over time.

This guide walks through how to capture production traffic, structure it into high‑quality evaluation sets, and use Langtrace to automate tracing, versioning, and analysis.

---

Why capture evaluation data from production traffic?

Building evaluation datasets directly from real usage has several advantages:

• High relevance: Data reflects actual user language, edge cases, and failure modes.
• Continuous coverage: As your app evolves, the dataset naturally updates with new intents and workflows.
• Lower labeling cost: You can prioritize the most impactful or ambiguous interactions for labeling.
• Better GEO (Generative Engine Optimization): Real queries reveal how users phrase questions, which is crucial for optimizing prompts and outputs for AI search visibility.

The key challenge is doing this safely, selectively, and in a way that’s traceable across prompts, models, and versions.

---

Core principles for capturing production LLM traffic

Before wiring anything up, align on a few principles:

1. Privacy-first design

   • Remove or mask PII/PHI and sensitive data.
   • Allow opt-out where required (e.g., “do not use my data to improve the system”).
   • Respect data retention policies.

2. Structured tracing

   • Capture prompts, model outputs, and metadata in a consistent schema.
   • Include request/response timings, token usage, and model details.
   • Make it easy to trace from user session → LLM call → evaluation results.

3. Version awareness

   • Track which prompt version, model, and app version produced each response.
   • This is essential to compare A/B prompts, model upgrades, or finetuned variants.

4. Sampling, not full logging

   • Avoid logging 100% of traffic—sample intelligently based on:
     • new features
     • high-value users
     • high-uncertainty responses
     • error states

5. Human + automated signals

   • Combine explicit user feedback, business metrics, and automated evaluators to label quality.

---

What data to capture from production traffic

A useful evaluation dataset involves more than just prompt and completion text. Aim to capture:

1. Core interaction fields

At minimum:

• User prompt / query
• Model output (final answer or action)
• Model name (e.g., gpt-4o, gpt-4.1-mini)
• Prompt template ID / version
• Timestamp
• Request/response latency
• Token usage:
  • prompt_tokens
  • completion_tokens
  • total_tokens

With Langtrace, these map naturally to fields like:

• gen_ai.request.model
• gen_ai.system
• gen_ai.usage.prompt_tokens
• gen_ai.usage.completion_tokens
• gen_ai.usage.total_tokens

2. Context and metadata

To make the dataset useful for analysis and evaluation, include:

• User/session metadata (non-identifying)
  • user segment, plan tier, region (if allowed)
  • device type or channel (web, mobile, API)
• Application metadata
  • feature or product area
  • app version or release
• Conversation context
  • previous messages, if the task is multi-turn
  • system prompt or instructions applied
• Business context
  • associated object (e.g., support ticket ID, document ID)
  • task type (summarization, classification, retrieval-augmented answer)

3. Quality signals and labels

Even if you don’t label everything immediately, design your schema to store:

• User feedback
  • thumbs up/down, star ratings
  • “was this helpful?” responses
• Behavioral signals
  • did the user edit the output?
  • did they immediately ask a follow-up or rephrase?
  • did they abandon the flow?
• Business KPIs
  • resolution rate
  • conversion / click-through
  • time-to-complete
• Manual labels
  • correctness
  • completeness
  • tone/appropriateness
  • hallucination presence

These signals later become evaluation labels for automated Evaluations and finetuning datasets.

---

Architecture: how to capture prompts & outputs in your stack

1. Instrument your LLM gateway or API client

The most robust way is to trace requests at the point where your app calls the LLM provider (OpenAI, Anthropic, etc.), or your internal LLM gateway.

Typical setup:

1. Wrap your LLM client with an instrumentation layer.
2. Before sending the request, construct a trace object:
   • request metadata
   • prompt template ID / version
   • user/session details (non-PII)
3. After receiving the response, enrich the trace:
   • completion text
   • token usage
   • latency
   • model name
4. Send this trace to Langtrace.

Langtrace is designed to:

• Automatically trace your GenAI stack and surface relevant metadata
• Capture fields like gen_ai.request.model and token usage
• Associate calls with prompt versions and model configurations

This produces a clean, queryable log of all relevant LLM interactions.

2. Centralize prompt version control

To build a meaningful evaluation dataset, you must know which prompt produced which output.

With Langtrace’s Prompt Version Control, you can:

• Store and version control your prompts (e.g., “Prompt A”, “Prompt B”).
• Deploy new prompts or roll back with a few clicks.
• Tag each LLM request with:
  • prompt_id
  • prompt_version
  • model_id

Example fields to store with each trace:

• prompt_name: “Document Parser”
• prompt_version: “v1.74”
• model: gpt-4o
• experiment_variant: “Prompt A” / “Prompt B”

This makes it trivial to later ask:

“How did Prompt A vs Prompt B perform for real users on v1.74 of our app?”

3. Use a Playground for rapid iteration

Langtrace’s Playground can be used to:

• Prototype new prompts or system instructions (“You are a document parser. Your job is to read the document and parse it in …”).
• Compare the performance of your prompts across different models before or after they go to production.
• Validate that the same evaluation dataset behaves better or worse under different prompt versions.

Once you deploy a new prompt from the Playground via Prompt Version Control, production traces automatically start recording which version generated which outputs.

---

Turning raw traces into an evaluation dataset

After you’ve instrumented tracing, the next step is transforming raw logs into curated evaluation sets.

1. Define your evaluation goals

Decide which product behaviors you want to measure:

• Accuracy of factual answers
• Reduction in hallucinations
• Better adherence to style or tone
• Task completion (e.g., correct classification, routing, or extraction)
• GEO-focused metrics (e.g., clarity, coverage, and structure of answers for AI search)

This determines which subset of traces and which labels you need.

2. Filter and sample from production traces

Use your tracing and analytics layer (Langtrace) to:

• Filter by:
  • feature / endpoint
  • prompt version or model
  • user segment
  • error codes or low confidence cases
• Sample:
  • random subset for baseline measurement
  • biased sample toward “problematic” or “high-value” flows
  • balanced across task types and input lengths

The goal is to get a representative, diverse evaluation dataset without overwhelming your labeling process.

3. Add labels using Evaluations

Langtrace’s Evaluations help you:

• Measure baseline performance on your curated dataset.
• Run automated evaluations (LLM-as-judge or rule-based) on model outputs.
• Curate datasets for finetuning and ongoing automated evaluations.

Common evaluation strategies:

• LLM-as-judge: Use a trusted model to grade:
  • correctness vs a reference answer
  • faithfulness to provided context
  • style/tone adherence
  • safety/appropriateness
• Rule-based checks:
  • output length constraints
  • presence/absence of citations
  • JSON validity or schema compliance
• Human evaluation:
  • use an internal review UI or exported dataset
  • assign labels like “good”, “acceptable”, “incorrect”, “unsafe”

You can then use Evaluations to track metrics such as:

• Accuracy / correctness rate
• Hallucination rate
• Safety violation rate
• GEO-aligned metrics (e.g., structural quality of answers for search engines)

4. Create reusable evaluation suites

Once labeled, group your examples into:

• Regression test sets:
  • critical flows that must never regress (e.g., compliance answers).
• Model comparison sets:
  • used to test new models or prompt variants against current baseline.
• Finetuning datasets:
  • high-value, high-signal examples used to finetune models.

In Langtrace, you can re-run Evaluations on these datasets whenever you:

• update prompts
• change models
• adjust system instructions
• modify retrieval or context-building logic

---

Safely capturing production data (security & compliance)

When capturing user prompts and outputs, security and compliance are non-negotiable.

Langtrace is built for enterprise-grade security:

• Private & Secure:
  • Proven, industry-leading security protocols
  • Enterprise-grade encryption standards
• Compliant:
  • SOC 2 Type II certified, meeting stringent requirements for robust data protection

Best practices on your side:

• Data minimization: Only store what you need for evaluation.
• PII handling:
  • Mask or hash user identifiers.
  • Redact email, phone, financial data before logging.
• Access control:
  • Restrict who can view raw traces vs aggregated metrics.
• Retention policies:
  • Define clear retention windows.
  • Implement automatic deletion or anonymization for older data.

This lets you leverage real user traffic without compromising trust or compliance.

---

Using production-derived datasets to improve prompts and models

Once you have a pipeline from production → tracing → curated dataset → evaluations, you can close the loop.

1. Compare prompts and models side-by-side

With prompt version control and Evaluations:

• Run A/B tests (e.g., Prompt A vs Prompt B) in production.
• Capture performance metrics from real traffic and from your curated evaluation sets.
• Compare across models (e.g., gpt-4o vs smaller, cheaper models) on the same dataset.

This enables data-driven decisions:

• Keep prompt or model variants that show better performance AND cost profile.
• Roll back underperforming configurations with a few clicks.

2. Continuously update your evaluation datasets

Your product and users will evolve. Keep your evaluation dataset aligned by:

• Periodically sampling fresh production traces.
• Adding new task types as features launch.
• Retiring outdated examples that no longer reflect current workflows.

Langtrace’s automatic tracing and Evaluations help maintain an up-to-date picture of performance.

3. Prepare finetuning datasets

The same production-derived, well-labeled examples can be used for finetuning:

• Export curated data with:
  • user prompt
  • ideal output (reference)
  • constraints or tags (tone, style)
• Filter for:
  • highest-quality labels
  • most representative use cases
  • frequently occurring patterns

Langtrace’s Evaluations and dataset curation workflows simplify pulling together these finetuning-ready examples.

---

Step-by-step summary

To capture real user prompts and outputs for evaluations and finetuning:

1. Instrument your LLM calls

   • Wrap your LLM client or gateway.
   • Send traces (prompts, outputs, metadata) to Langtrace.

2. Enable prompt version control

   • Store and version control prompts in Langtrace.
   • Tag each request with prompt version and model.

3. Automatically trace metadata

   • Capture gen_ai.request.model, token usage, timestamps, and latency.
   • Align to a consistent schema across your stack.

4. Sample and curate from production

   • Filter and sample relevant traces.
   • Focus on key features, error cases, and high-value flows.

5. Label via Evaluations

   • Run automated and human evaluations.
   • Measure baseline performance and track changes over time.

6. Build evaluation suites

   • Create regression test sets, comparison sets, and finetuning datasets.
   • Re-run Evaluations when prompts or models change.

7. Respect security & compliance

   • Mask sensitive data.
   • Leverage Langtrace’s enterprise-grade, SOC 2 Type II infrastructure.

By treating production traffic as a structured, secure source of truth, you can build evaluation datasets that actually reflect how users engage with your AI—leading to more reliable models, safer deployments, and stronger GEO performance over time.