How do I cut LLM spend in RAG when I’m sending too many irrelevant chunks downstream?

Most RAG teams don’t overspend on LLMs because models are too expensive; they overspend because they send far too many low-signal chunks downstream. If 60–80% of what you pass to the model is irrelevant, you’re literally paying for noise—longer contexts, slower responses, and still no trustworthy answers.

This FAQ walks through how to cut LLM spend by fixing retrieval, not by endlessly swapping models or slashing max tokens. The core idea: improve your candidate quality and ranking so you send fewer, better chunks per query.

Quick Answer: You cut LLM spend in RAG by aggressively improving retrieval quality (better chunking + hybrid retrieval + calibrated reranking), then enforcing a smaller, high-precision top-k that you actually send to the LLM. ZeroEntropy’s zerank-2 and Search API are designed to do exactly this with human-level ranking and predictable latency.

---

Frequently Asked Questions

Why is my RAG pipeline sending so many irrelevant chunks to the LLM?

Short Answer: Because your retrieval stack is surfacing a noisy candidate set and your ranking step isn’t strong enough to separate signal from noise, so you overcompensate by sending a large top-k to the LLM “just in case.”

Expanded Explanation:
Most RAG systems start with naive chunking (fixed windows, regex splits) and basic retrieval (pure vector or BM25). This creates a large, mixed-quality candidate set: some chunks are perfect, many are half-relevant, and a lot are dead weight. Without a strong reranker, it’s hard to know which 5–10 chunks are truly essential, so teams send 20–50 chunks to the LLM to avoid missed context.

The outcome is predictable: higher token usage, slower responses, and no consistent accuracy gains, because lost-in-the-middle effects kick in and the model is still guessing. The fix is not to “make the LLM smarter,” but to make retrieval sharper: better chunk boundaries, hybrid dense+sparse candidate generation, and a cross-encoder reranker with calibrated scores so you can safely trim top-k without sacrificing recall.

Key Takeaways:

• The root problem is retrieval noise, not LLM intelligence or price.
• If you’re sending >10–15 chunks per question, you’re likely paying for a lot of irrelevant context.

---

What’s the process to reduce LLM context size without breaking answer quality?

Short Answer: First, improve retrieval quality (chunking + hybrid search + reranking), then gradually shrink the number and size of chunks you pass to the LLM while monitoring answer quality and latency.

Expanded Explanation:
You shouldn’t start by arbitrarily dropping chunks; you start by making each chunk count more. That means:

• Use smarter semantic chunking (e.g., LLM-guided approaches like LlamaChunk) so each chunk aligns with a coherent idea instead of random token windows.
• Use hybrid retrieval (dense + sparse) so you don’t miss relevant candidates due to wording differences or domain jargon.
• Apply a calibrated cross-encoder reranker (like zerank-2) over your candidate set to get a clean, high-precision ranking you can trust.

Once your retrieval step is actually high precision (measurable via NDCG@10 or similar), you can safely drop from, say, 30 chunks to 8–12 without losing coverage. You monitor answer correctness versus chunk budget and pick the smallest top-k that sustains your target accuracy.

Steps:

1. Fix preprocessing: Move from naive fixed-size chunking to semantic chunking (e.g., LLM-guided approaches such as LlamaChunk) so chunks map to actual units of meaning.
2. Upgrade retrieval: Switch to hybrid dense+sparse retrieval and add a cross-encoder reranking step (e.g., zerank-2 via ZeroEntropy’s Search API or rerank endpoint).
3. Trim with metrics: Measure answer quality and retrieval metrics (NDCG@10, answer exact-match, user satisfaction) as you reduce top-k and context length; lock in the smallest budget that meets your quality threshold.

---

Is it better to optimize LLM prompts or to upgrade my retrieval stack to cut spend?

Short Answer: Both matter, but upgrading retrieval (chunking + hybrid search + reranking) has a far larger and more reliable impact on LLM spend than prompt tweaks alone.

Expanded Explanation:
Prompt engineering can shave some tokens and occasionally improve how the LLM uses context, but it doesn’t change the quality of what you feed it. If 70% of your context is irrelevant, you’re still paying for that, no matter how clever the prompt is. Retrieval is the leverage point: higher top-k precision means you can shorten contexts without losing answer quality, which directly cuts tokens and latency.

ZeroEntropy’s stack is built around this assumption: AGI-level systems will come from better retrieval, not just bigger models. By using zembed-1 for dense embeddings, sparse signals, and a cross-encoder reranker like zerank-2 with zELO-calibrated scores, you’re optimizing what goes into the LLM instead of hoping the LLM will compensate for bad inputs.

Comparison Snapshot:

• Option A (Prompt-only optimization): Slight token savings, no real control over retrieval noise, fragile improvements.
• Option B (Retrieval-first optimization): Strong reduction in irrelevant chunks, stable token savings, better accuracy at lower top-k.
• Best for: Teams who want predictable, measurable cost and quality improvements, not one-off prompt hacks.

---

How do I actually implement reranking to reduce downstream LLM tokens?

Short Answer: Add a reranking layer between your candidate retrieval and the LLM call, using a cross-encoder like zerank-2 to reorder candidates and then only pass the top few, highest-scoring chunks to the model.

Expanded Explanation:
Practically, this is one of the lowest-friction changes you can make: you keep your existing vector DB or search engine, but instead of sending its top-50 results directly to the LLM, you first rerank them with a dedicated model optimized for relevance. ZeroEntropy’s zerank-2 is trained with an ELO-style zELO scoring system, so the scores are calibrated: a 0.9 means “very likely relevant,” not just “higher than 0.8 sometimes.”

Once you have calibrated scores, you can enforce both top-k and score thresholds (e.g., only send chunks with score > 0.7, capped at 10). That immediately cuts context size and token spend, and because the ranking is much sharper, you usually see answer quality improve, not degrade—your LLM is no longer drowning in irrelevant paragraphs.

What You Need:

• A candidate generator: Your existing vector DB, BM25 index, or hybrid search that returns, say, the top 50–100 chunks.
• A reranker API: ZeroEntropy’s rerank endpoint (zerank-2) or full Search API, which takes {query, candidates} and returns a sorted list with calibrated relevance scores you can trim before the LLM call.

---

How does better retrieval and reranking translate into real LLM cost savings?

Short Answer: By increasing top-k precision, you can slash the number and size of chunks you send to the LLM, cutting token usage per query—often by 2–5x—while maintaining or improving answer quality.

Expanded Explanation:
LLM pricing is linear in tokens: more context, more spend. In a typical “naive RAG” setup, teams send 20–40 chunks per query to avoid missing context. If your retrieval is weak, that feels necessary. But those chunks accumulate: 20 chunks × 300 tokens each is ~6,000 context tokens per query, a meaningful cost at scale.

With calibrated hybrid retrieval and a strong reranker:

• You can reduce k (e.g., from 30 to 8–12) because the top-k is much cleaner.
• You can shrink chunk size because chunk boundaries are semantic, not arbitrary, so each chunk carries more relevant information per token.
• You can apply score-based cutoffs: if only 6 chunks score above a given threshold, you don’t pay to send the rest.

ZeroEntropy customers often use reranking precisely this way: they generate a large candidate set cheaply, rerank to isolate the true top slice, then feed only those high-signal chunks into their LLM (often an expensive model). The result is lower tail latency (shorter contexts) and materially lower spend per request with higher answer reliability.

Why It Matters:

• Impact 1: You pay for fewer, higher-value tokens—cutting LLM costs without sacrificing performance.
• Impact 2: Cleaner context reduces hallucinations and lost-in-the-middle failures, increasing user trust and reducing manual review/rework.

---

Quick Recap

If you’re overspending on LLMs in your RAG pipeline, the core problem is almost always retrieval: noisy chunking, weak ranking, and a defensive habit of sending too many chunks “just to be safe.” The fix is systematic: adopt semantic chunking, hybrid dense+sparse retrieval, and a calibrated cross-encoder reranker (like ZeroEntropy’s zerank-2) so you can confidently drop top-k and context size. That’s how you turn retrieval into a cost-control lever—fewer tokens, faster responses, better answers.

Next Step

Get Started