LLM Observability & Evaluation
Why are our RAG answers suddenly less accurate even though we didn’t change the code?
7 min read
Most teams hit a moment where their RAG system “just gets worse” without a single code change. Nothing shipped, no new model version (as far as you know), yet users suddenly complain about irrelevant or hallucinated answers. When you’re running agentic systems in production, this isn’t a mystery; it’s drift—across your data, retrieval, or model stack—showing up as silent failures.
Quick Answer: RAG accuracy often drops due to hidden changes in data, embeddings, retrieval configuration, or upstream models—not your application code. Without end-to-end traces and ongoing evaluations, these shifts show up as “random” degradation instead of clearly observable drift you can debug and fix.
Frequently Asked Questions
What usually causes RAG answers to get worse when the code hasn’t changed?
Short Answer: Accuracy typically drops because something around your code changed—data, embeddings, indexes, models, or prompts—not the application logic itself.
Expanded Explanation:
RAG pipelines are multi-component systems: ingestion jobs, embeddings, vector stores, rankers, LLMs, prompts, and sometimes tools/agents. Any change or drift in one of these layers can degrade answer quality while your codebase stays untouched. Common culprits include stale or partially updated indices, a silently upgraded base model, modified retrieval parameters (e.g., top_k, filters), or new content that confuses retrieval.
In production, these failures appear as inconsistent behavior: some queries still look fine while others suddenly miss critical context or hallucinate. Without distributed traces that show which documents were retrieved, which embeddings were used, and which model actually answered, you’re left guessing. Teams that treat RAG as a black box feel like “the same code randomly got worse,” when in reality the underlying system shifted.
Key Takeaways:
- “No code change” doesn’t mean “no change”—data, embeddings, models, and configs all drift.
- You need end-to-end observability (traces + evals) to see where accuracy is actually degrading.
How do I systematically debug why my RAG answers are less accurate?
Short Answer: Start by tracing specific bad answers end-to-end—retrieval, ranking, prompts, and model calls—then check for data, embedding, or model drift before you touch application logic.
Expanded Explanation:
Treat each bad answer like a failed request in traditional software. You want to replay the exact execution path: what query the user sent, which documents were retrieved (and why), how the prompt was constructed, and which model produced the final response. With HoneyHive’s Traces, you get distributed spans across the entire RAG pipeline, including embedding calls, vector queries, and LLM generations.
Once you can see the full trace, you can classify the failure mode: missing context (retrieval issue), incorrect interpretation of correct context (LLM or prompt issue), or outdated/incorrect source data (ingestion/indexing issue). From there, you can build targeted tests and evaluations so the same pattern can’t silently regress again.
Steps:
-
Capture and inspect traces for failing sessions
Use OpenTelemetry-native tracing to log spans for user query → embedding → vector search → reranking (if any) → LLM call → final answer. Confirm which docs were actually retrieved. -
Label the failure mode using evaluations
Run automated evaluators like Context Relevance and Answer Faithfulness on those traces, then add human annotations for nuanced cases (e.g., “partial context,” “wrong doc, right answer,” “hallucination”). -
Turn failures into a regression suite
Convert failing traces into a HoneyHive Dataset and run Experiments with different retrieval configs, models, or prompts. Add regression checks so you catch similar issues in CI/CD before they hit production again.
Is this a retrieval problem, a model problem, or a data problem?
Short Answer: Retrieval issues show up as missing or irrelevant documents; model issues show up as misusing otherwise good context; data issues show up as outdated or conflicting source content.
Expanded Explanation:
You can usually differentiate the root cause by inspecting the retrieved context and comparing it to the question and answer. HoneyHive’s Session Replays and Traces make this concrete: you see the query, you see the retrieved snippets, and you see the final answer in one timeline.
- If the correct information isn’t in any retrieved chunk, you have a retrieval/indexing problem (embeddings, filters,
top_k, index freshness). - If the correct information is present but the answer is wrong or hallucinated, you likely have a model/prompt problem (instruction quality, model drift, tool misuse).
- If the context is technically correct but outdated or contradictory, it’s a data/ingestion problem (stale documents, inconsistent schemas, missing deletions).
Comparison Snapshot:
-
Retrieval Problem: Correct info missing from retrieved docs; embeddings, vector index, or search filters changed or drifted.
-
Model/Prompt Problem: Correct info present in docs, but the answer is wrong, incomplete, or unsafe; often tied to model changes or prompt regressions.
-
Data Problem: Retrieved docs are outdated, duplicated, or conflicting; ingestion jobs or source systems changed over time.
-
Best for: Use retrieval-focused fixes when context is wrong/absent, model/prompt fixes when context is good but answers are bad, and data pipeline fixes when content itself is stale or broken.
How can we monitor and prevent RAG accuracy from drifting silently?
Short Answer: Put your RAG system under continuous evaluation—online and offline—so you track answer quality over time alongside latency and cost, then set alerts on drift.
Expanded Explanation:
Production RAG systems need the same rigor as any critical API: metrics, monitors, and regression checks. Instead of just tracking latency and error codes, you also measure semantic quality with automated and human evaluations. In HoneyHive, you can run Online Evaluation on live traffic, scoring each span or session with evaluators like Answer Faithfulness, Context Relevance, and Tool Misuse.
Once you’re generating quality scores, you can attach Monitors and Alerts. For example, trigger an alert if average Context Relevance for a given index or tenant drops beyond a threshold, or if a specific model’s Answer Faithfulness regresses after a deployment. When a monitor fires, you can automatically route traces to Annotation queues or add them to regression datasets, turning live failures into test cases.
What You Need:
- Online + offline evaluations wired into your traces (automated evaluators + human review).
- Alerts and drift detection on key quality metrics (relevance, faithfulness, safety) tied to regression workflows.
How should we adapt our RAG strategy so accuracy stays high as the system evolves?
Short Answer: Treat RAG as an evolving system: continuously curate datasets from production, experiment with retrieval and model configs, and bake regression checks into CI/CD so you ship changes with measurable quality guarantees.
Expanded Explanation:
A RAG stack that’s “good enough today” will drift tomorrow as content grows, user queries diversify, and underlying models change. The teams that maintain high accuracy don’t rely on ad-hoc debugging; they close the loop from production traces → datasets → experiments → CI. HoneyHive is designed around this loop.
Start by converting representative production traces (including failures) into Datasets, then run Experiments anytime you adjust indexing strategies, embedding models, or prompts. Use Custom Evaluators—LLM-as-a-judge or code-based—to encode your business-specific definition of “good,” and back it up with Human Evaluators for high-risk flows. Finally, plug Regression Detection into your release process so any change to retrieval logic, embeddings, or models is automatically vetted on real-world test cases before it goes live.
Why It Matters:
- Prevent silent quality regressions when you update indexes, swap models, or change prompts—especially in mission-critical agentic systems.
- Align the system with domain experts by capturing their feedback as reusable evaluators and datasets instead of one-off comments or tickets.
Quick Recap
When RAG answers suddenly get less accurate even though you didn’t change the code, something else in the system changed: your data, embeddings, retrieval configuration, or models. The fix isn’t guesswork; it’s observability and evaluation. Use distributed traces to see exactly what changed in each failing request, run targeted evaluators (context relevance, faithfulness, safety) to classify failure modes, and convert real production failures into datasets and regression tests. With continuous online evals, alerts, and experiments wired into your stack, you can catch drift early and keep RAG accuracy stable as your system scales.