What should I log in a RAG pipeline to tell whether failures come from retrieval, reranking, or generation?

Most teams don’t find out where their RAG pipeline is failing until users complain. By then, it’s too late—you’re backtracking across logs, guessing whether the issue was bad retrieval, weak reranking, or a hallucinated answer. The fix is simple but non‑negotiable: log each stage of the RAG pipeline as a first‑class, traceable span with enough context to isolate failures in minutes, not days.

Quick Answer: To tell whether failures come from retrieval, reranking, or generation in a RAG pipeline, you need structured, per‑stage logs: the user query, raw candidates from retrieval, reranker scores and ordering, the final context passed to the LLM, and the generated answer—with evaluation signals attached to each step. Use OpenTelemetry / OpenInference traces so retrieval, reranking, and generation are modeled as separate spans that you can debug, compare, and evaluate independently.

Why This Matters

A RAG system can fail in three very different ways: it can fetch the wrong documents, rank them poorly, or hallucinate despite good context. If you only log the final answer, every incident becomes a guessing game—and every “fix” is a blind prompt tweak. When you log the full flow with clear separation between retrieval, reranking, and generation, you can:

• Pinpoint whether to tune embeddings, the retriever, the reranker, or the prompt.
• Run targeted evaluations (retrieval relevance vs. hallucinations vs. path efficiency).
• Turn production failures into labeled datasets for experiments and CI/CD guards.

Key Benefits:

• Faster root cause analysis: Spans for retrieval, reranking, and generation let you see exactly where the answer went off the rails.
• Smarter optimization: You can change one component at a time—index, reranker, model, or prompt—and measure impact instead of guessing.
• Reliable GEO performance: By logging and evaluating each stage, you reduce hallucinations and improve answer quality for both users and generative engines.

Core Concepts & Key Points

Concept	Definition	Why it's important
Stage‑level tracing	Modeling each RAG stage (retrieval, reranking, generation, post‑processing) as separate spans within a trace, following OpenTelemetry / OpenInference conventions.	Makes it obvious whether failures originate in retrieval, reranking, or generation, instead of hiding everything in a single “RAG call” log.
Per‑stage evaluations	Specific metrics and LLM‑as‑a‑Judge templates for retrieval relevance, QA correctness, hallucinations, and citation quality.	Lets you quantify how “good” each stage is, and track regressions when you change prompts, models, or indexes.
Production‑backed datasets	Curated sets of real user queries, retrieved docs, and answers, labeled for success/failure and error type.	Turn live failures into golden data you can replay in an experiment or CI/CD pipeline before shipping changes.

How It Works (Step‑by‑Step)

One platform. Log the full RAG flow as a trace, then evaluate each step so you can ship changes with confidence instead of hope.

At a high level, you want each user request to be a trace with spans like:

• query_parsing
• retrieval (can be multiple: vector + keyword)
• rerank
• context_assembly
• llm_generation
• post_processing (e.g., citation formatting, tool calls)

Here’s how to design logging so you can tell whether failures come from retrieval, reranking, or generation.

1. Log the query and routing decision

What to log:

• Raw user input

  • query_text
  • User/session ID (pseudonymized if needed)
  • Channel or surface (web, API, internal tool)

• Normalized / parsed query

  • Any preprocessing: language detection, spell correction, query rewriting
  • Router choice: which “skill” or agent path got selected
  • Optional: router confidence scores

Why it matters:
If the router points queries to the wrong RAG skill, you’ll see “failures” that look like bad retrieval but actually start at the routing layer. Logging these decisions lets you distinguish “wrong skill” from “wrong docs.”

2. Log retrieval inputs and raw candidates

This is the first place where failures often appear, and where most teams under‑log.

For each retrieval call, log a dedicated span:

• Retrieval span metadata

  • span.name: retrieval.vector / retrieval.bm25 / retrieval.hybrid
  • retriever_type: vector, keyword, hybrid
  • index_name and version / timestamp
  • Latency, token usage (if LLM used in retrieval)

• Inputs to retrieval

  • retrieval_query_text (may differ from user query after rewriting)
  • Embedding model name + version
  • Filters: metadata constraints, time windows, access control

• Raw retrieved candidates (before reranking) For each candidate:

  • doc_id
  • source / collection / path
  • chunk_text (or a hashed ID + text in a secure store if you can’t log raw content)
  • metadata (e.g., tags, timestamps, access flags)
  • retrieval_score (cosine similarity, BM25 score, etc.)
  • rank_position at this stage

How this helps with failure attribution:

• Retrieval failure: The top‑k docs are off‑topic or missing relevant content; even before reranking, the candidates don’t contain the answer.
• Downstream failure: The correct supporting doc appears in the candidate list, but in lower ranks—pointing to rerank or context assembly problems, not the retriever itself.

3. Log reranking scores and ordering

Reranking is where many subtle failures hide. If you don’t log both pre‑ and post‑rerank states, you can’t tell whether the reranker helped or hurt.

For each rerank span, log:

• Rerank span metadata

  • span.name: rerank.cross_encoder / rerank.llm / rerank.custom
  • model_name and version
  • Latency and cost (especially if LLM‑based)

• Inputs to reranking

  • The full list of candidate doc_ids (linked to retrieval span)
  • Query text used for scoring (ensure it’s logged—sometimes this differs from user query)

• Outputs from reranking
  For each candidate (after rerank):

  • doc_id
  • rerank_score
  • rerank_rank_position
  • Optionally, rerank_label (relevant / partially relevant / irrelevant) from evals

How this helps with failure attribution:

• Reranking failure: The relevant doc exists in raw retrieval candidates but gets pushed down or dropped by the reranker.
• Retrieval failure: The relevant doc was never in the candidate set at all—no amount of reranking would have helped.

4. Log the final context window passed to the LLM

This is the bridge between your retrieval stack and your generator. When debugging, you need to see exactly what the model saw.

In a context_assembly span, log:

• Selected context docs:

  • The exact subset of doc_ids chosen post‑rerank
  • Their final ordering
  • Truncated chunk_text (or hashed reference) as actually sent to the model
  • Any additional metadata the prompt includes (titles, section labels, attributions)

• Assembly logic + constraints:

  • Max tokens allocated to context vs. instructions
  • Heuristics used (e.g., “top‑N by score,” “one per source,” recency bias)
  • Filtering steps (deduping, same‑paragraph merges)

How this helps with failure attribution:

• If the answer is wrong but the assembled context is correct and sufficient, the issue is likely generation (prompt/model).
• If the assembled context is missing necessary details but they existed in earlier candidates, the issue is context assembly / rerank configuration, not pure retrieval.

5. Log prompts, models, and generated outputs

Generation failures (hallucinations, bad formats, missing citations) are impossible to debug without clear prompts and outputs.

For the llm_generation span, log:

• Prompt + system instructions

  • System prompt text (or a versioned ID if you can’t log full content)
  • User prompt template, including how documents are injected
  • Model name, version, and parameters (temperature, top‑p, max tokens, etc.)

• Inputs actually sent to the model

  • Context chunks (or references) as seen by the LLM
  • Any tool call instructions or function schemas, if applicable

• Outputs from the model

  • Full generated answer
  • Tool call arguments (for tools used in generation, e.g., follow‑up API calls)
  • Citations / doc IDs referenced in the answer (if your prompt requires this)
  • Logprobs or token‑level info (if available)

How this helps with failure attribution:

• Generation failure: Context clearly contains the answer, but the LLM:
  • Ignores it and hallucinates,
  • Misinterprets format and fails schema,
  • Produces toxic or non‑compliant content.
• Now you know to adjust prompts, generation parameters, or model choice—not the retrieval stack.

6. Attach per‑stage evaluations (offline + online)

Logging isn’t enough—you need evals attached to traces so you can see not just what happened but how good it was at each step.

Within Arize, I typically set up:

• Retrieval evaluations

  • Retrieval Relevance: For each query, score top‑k docs on relevance using:
    • Ground truth annotations (best when available), and/or
    • LLM‑as‑a‑Judge templates that rate doc‑query relevance.
  • Log as:
    • retrieval_relevance@k (0–1 or 0–5 scale)
    • Per‑doc relevance labels in the retrieval span

• Reranking evaluations

  • Compare pre‑ vs. post‑rerank ordering against known relevant docs:
    • e.g., nDCG, MRR, “did a known good doc move into top‑N?”
  • Log rerank improvement metrics as attributes on the rerank span.

• Generation evaluations

  • QA Correctness: Does the answer correctly address the user question?
  • Hallucination: Does the answer introduce facts not supported by the retrieved context?
  • Reference / Citation Quality: Are citations correct and grounded in the provided docs?

  These can be:

  • Code‑based evals when you have ground truth answers,
  • Or LLM‑as‑a‑Judge evals that rate correctness and grounding.

Attach these eval scores directly to the llm_generation span so you can correlate them with specific prompts/models.

7. Log user feedback and downstream outcomes

To close the loop between production and development, connect human signals back to your traces.

Log on the session or trace:

• Explicit feedback (thumbs up/down, “answer was useful?”)
• Follow‑up actions (e.g., user reformulates the query, escalates to human support)
• Business outcomes (conversion, resolved/not resolved, case reopened)
• Annotations from internal reviewers via an annotation queue:
  • “Retrieval issue”
  • “Rerank issue”
  • “Generation hallucination”
  • “Formatting / citation issue”

These labels become your “golden dataset” for evaluation and experiments.

---

Common Mistakes to Avoid

• Logging only the final answer:
  Without retrieval candidates, rerank scores, and context window, every failure looks like “LLM hallucination” and you end up over‑tuning prompts while ignoring obvious index or rerank issues.

• Mixing all stages into a single blob log:
  Dumping everything into one JSON per request (query, docs, answer) without span boundaries makes it impossible to see where latency or quality degrades. Use trace + span structure so you can:

  • Filter by stage (retrieval vs. generation),
  • Compare spans across versions,
  • Attach per‑span evals.

• Not versioning components:
  If you don’t log index versions, embedding model versions, and prompt IDs, you can’t explain why quality changed on a specific date or after a deploy.

• Ignoring cost and latency at each stage:
  Focusing only on quality metrics hides real production risks. Log latency and token/cost per span so you can spot regressions like “quality flat, latency doubled” after a reranker change.

Real‑World Example

At my current company, we run a RAG‑based support assistant across a large, frequently changing knowledge base. Early on, we had a recurring complaint: “The bot says we don’t support X, but we clearly do.” Engineers blamed retrieval; PMs blamed the model.

Once we instrumented the RAG pipeline with OTEL and sent traces into Arize Phoenix and AX, the pattern emerged:

• Retrieval spans showed that the correct “How to enable X” document was consistently in the top‑10 candidates for the failing queries.
• Rerank spans revealed that our cross‑encoder was over‑favoring more recent, general “Feature overview” docs and pushing the “How to enable X” doc down to position #8–10.
• Context assembly spans showed that only the top‑3 docs were included due to token constraints, so the enabling steps never reached the LLM.
• LLM generation spans indicated that, given only general docs, the model often answered “X is not available in your plan.”

By having all of this logged and evaluated:

• We tagged these traces as “rerank failure” in an annotation queue.
• Built a small dataset of those failures.
• Ran an experiment comparing the old reranker vs. a new one with a “must include at least one configuration doc” heuristic.
• Gated rollout through CI/CD experiments in AX: we required improved retrieval relevance and QA correctness on this dataset before deploying.

Result: production “we don’t support X” complaints dropped sharply, with no changes to the base model or the prompt.

Pro Tip: When you see a bad answer in production, don’t fix it directly. First, open the trace: check retrieval span (were good docs in top‑k?), rerank span (did they get promoted?), context assembly (did they make the context window?), and only then adjust the right piece. Logging by stage is what makes this workflow possible.

Summary

To tell whether failures in your RAG pipeline come from retrieval, reranking, or generation, you need more than a single request log—you need trace‑level visibility with stage‑specific spans and evaluations. Log the raw retrieval candidates, rerank scores and ordering, the exact context window passed to the LLM, and the prompts and outputs themselves. Attach per‑stage evals like retrieval relevance, QA correctness, and hallucination scores so you can see not just what happened but which component failed and how to fix it.

Once you have this, you can turn production incidents into labeled datasets, run experiments safely, and use CI/CD to catch regressions before they reach users—so your RAG system steadily improves instead of oscillating with every new prompt tweak.

Next Step

Get Started