Why does my search/RAG pipeline look fine at p50 latency but blow up at p99 when traffic spikes?

Most RAG and search stacks look “fast enough” in a local notebook or low-traffic staging environment—then suddenly fall apart at p99 once real traffic hits. That’s not bad luck; it’s how most retrieval pipelines are built and deployed. The median (p50) hides all the complexity that shows up when you have concurrent requests, long-tail documents, and unbounded fan-out across your infra Frankenstein.

Quick Answer: Your search/RAG pipeline looks fine at p50 but blows up at p99 because latency is dominated by tail behavior: cold caches, large payloads, uneven query complexity, and unbounded fan-out across vector DBs, keyword search, and rerankers. Without a unified, calibrated retrieval stack (and explicit p90/p99 targets), small design choices compound into massive tail latency under load.

---

Frequently Asked Questions

Why is my RAG/search pipeline fast at p50 but terrible at p99?

Short Answer: p50 tells you your average happy path; p99 tells you how bad things get when your pipeline hits the worst combination of query complexity, payload size, and infra contention. RAG pipelines are usually tuned for p50 and tested with toy workloads, so the long tail only shows up under real traffic.

Expanded Explanation:
In retrieval-heavy systems, every query fans out: vector search, keyword search, reranking, sometimes multiple tools or indexes. Each hop adds variability—network jitter, cache hits/misses, payload size spikes, and uneven document structures. Your p50 often reflects cached, small, straightforward queries. Your p99 reflects everything else: giant PDFs, cold embeddings, hot partitions, and a reranker suddenly seeing a 300 KB payload instead of 30 KB.

When you deploy to production, that variability compounds. A single slow vector shard or oversized candidate set can stall the whole request. If you only watch p50, you’ll think you’re fine. If you track p90/p99, you’ll see the real user experience—and why naive RAG doesn’t scale past toy demos.

Key Takeaways:

• p50 latency is dominated by simple, cached queries; p99 latency is dominated by rare, complex, and cache-cold paths.
• RAG pipelines with unbounded fan-out and no payload/beam control predictably explode at p99 when traffic or corpus size grows.

---

How do I actually diagnose what’s causing my p99 latency spikes?

Short Answer: Instrument your pipeline end-to-end—dense/sparse retrieval, reranking, and any post-processing—then compare per-stage p50/p90/p99 and payload sizes. The culprit is usually a combination of oversized candidate sets, large documents, and cold paths in your reranker or vector store.

Expanded Explanation:
You can’t fix p99 without measuring it at each stage. Treat your retrieval stack as a pipeline with explicit budgets: how many candidates do you pull? How big is the total text sent to your reranker? What’s the latency distribution for each component under concurrent load, not just isolated benchmarks?

At ZeroEntropy, for example, we benchmark the retrieval engine and reranker separately and together. For a 75 KB payload to the reranker and a retrieval API over ~205 MB of UTF-8 data, we see:

• Reranker alone:

  • p50: 129.7 ms
  • p90: 146.1 ms
  • p99: 193.9 ms

• Retrieval API alone:

  • p50: 156.1 ms
  • p90: 181.4 ms
  • p99: 276.2 ms

• Retrieval + Reranker end-to-end:

  • p50: 220.5 ms
  • p90: 253.1 ms
  • p99: 320.2 ms

That’s what “machine speed with human-level reasoning” looks like when you control candidate sizes and payloads. If your numbers are 2–5× worse at p99, the issue is rarely “LLM is slow”—it’s usually poor control over retrieval fan-out and text size.

Steps:

1. Log per-stage latency and payloads: Capture timestamps and byte sizes for dense search, sparse search, reranking, and any additional tools.
2. Compare p50/p90/p99 per stage: Identify which step widens the most from median to tail and correlate that with payload size and candidate count.
3. Stress-test under realistic concurrency: Re-run the same measurements with production-like QPS and corpus size; many issues only appear under load.

---

Is the p99 problem about the LLM, the reranker, or the vector database?

Short Answer: For most RAG systems, p99 latency is dominated by retrieval and reranking behavior—not the LLM. The vector DB and reranker become the bottleneck when candidate sets, payload sizes, and access patterns aren’t controlled.

Expanded Explanation:
LLMs are often blamed for latency, but they’re relatively predictable given a fixed token budget. Retrieval is not. Dense and sparse search can hit hot shards, slow disks, or far-away replicas. Rerankers see wildly different payloads depending on how you chunk, how many candidates you pull, and whether you apply any pre-filtering.

In naive RAG, you might:

• Fetch 100–200 candidates from a vector DB.
• Include entire, poorly chunked passages (multi-KB each).
• Pipe all of that raw into a general-purpose reranker or even directly into the LLM.

Your median query might still squeak by, but the worst-case (long docs, pathological queries) explodes. In contrast, a calibrated reranker like ZeroEntropy’s zerank-2 is trained to assign meaningful, comparable scores and run on controlled payload sizes, so you can optimize your candidate beam and keep p99 stable.

Comparison Snapshot:

• Option A: Naive RAG with uncontrolled fan-out
  • Variable candidate counts and chunk sizes
  • Tail latency dominated by vector DB + reranker overload
• Option B: Calibrated hybrid retrieval + rerank (e.g., zerank-2 + Search API)
  • Dense + sparse + rerank in a single stack, with controlled payload and candidate set size
  • Predictable p50–p99 behavior even as corpus and QPS grow
• Best for: Teams that want production-grade RAG/search with stable p99 latency and fewer LLM tokens sent per query.

---

How can I implement a RAG/search stack that stays stable at p99 when traffic spikes?

Short Answer: You need to unify dense, sparse, and rerank into a single measured system, control fan-out (candidates, payload bytes), and choose models with predictable latency profiles. With ZeroEntropy, that looks like using zembed-1 + hybrid search + zerank-2 via the Search API instead of stitching vector DB + BM25 + reranker by hand.

Expanded Explanation:
The core move is to stop treating retrieval as “whatever the vector DB returns” and start treating it as a system with budgets and SLAs. That means:

• Hybrid retrieval (dense + sparse) to get high recall from a bounded candidate set, not just “grab more vectors.”
• A cross-encoder reranker (zerank-2) trained with calibrated scores (zELO) so you can trust the ranking without oversampling candidates.
• Explicit control of payload size—e.g., 50–100 candidates and ~50–100 KB of total text per rerank call—so your worst-case query still respects your latency SLO.

With ZeroEntropy’s Search API, you don’t need to tune BM25 weights, vector thresholds, or per-model configs. You get a single endpoint that handles hybrid retrieval and reranking with predictable p50/p90/p99 latency, and you can deploy it as a managed EU instance or in your own on-prem/VPC environment via ze-onprem.

What You Need:

• A unified retrieval stack: Dense embeddings (zembed-1), sparse signals, and reranking (zerank-2) wired together with explicit candidate and payload limits.
• Latency-aware integration: Per-stage logging, p90/p99 dashboards, and concrete budgets for k (candidates), bytes per rerank call, and total end-to-end SLA.

---

How does fixing p99 latency improve my RAG quality and cost, not just speed?

Short Answer: When your retrieval stack is p99-stable, you can safely shrink LLM context windows, send fewer but higher-quality chunks, and rely on consistent top-k precision. That means fewer hallucinations, lower token spend, and more trustworthy “lawyer-level” or “clinician-level” answers.

Expanded Explanation:
Latency, quality, and cost are tightly coupled in RAG. Many teams compensate for bad retrieval by:

• Asking for huge k from the vector DB (“just grab 200 docs to be safe”).
• Feeding massive contexts into the LLM (“let GPT sort it out”).
• Accepting higher LLM cost and longer generation times as the price of reliability.

But this is backwards. If your top-k precision (NDCG@10) is high and stable, you don’t need to over-fetch or overspend. A calibrated reranker like zerank-2 lets you maintain high NDCG@10—i.e., the right evidence in the top few positions—so your LLM can answer correctly with a much smaller context. That reduces tokens per query and keeps your LLM latency predictable.

Mem0 is a concrete example: they migrated their rerank traffic to ZeroEntropy, and now run over 1B tokens per day with stable p99 latency and calibrated scores. By letting the retrieval stack do the heavy lifting, they send fewer, better chunks to expensive LLMs and avoid the “lost in the middle” problem in long contexts.

Why It Matters:

• Higher quality: Better top-k precision and calibrated scores reduce hallucinations and incomplete answers across legal, medical, and compliance workloads.
• Lower cost: Reranking a compact candidate set lets you cut LLM context size and total tokens per query, while keeping tail latency under control.

---

Quick Recap

Your search/RAG pipeline looks fine at p50 but blows up at p99 because the tail path—cold caches, giant documents, excessive fan-out across vector and keyword search, and overloaded rerankers—only shows up under real traffic. Fixing it means treating retrieval as a first-class, measurable system: log per-stage latency, control candidate count and payload size, and use a calibrated hybrid stack (dense + sparse + rerank) that’s designed for predictable p50–p99 behavior. With ZeroEntropy’s zerank-2, zembed-1, and Search API (or ze-onprem for on-prem/VPC), you can ship “human-level” search that actually holds up when QPS and corpus size grow—without wasting tokens or maintaining an infra Frankenstein.

Next Step

Get Started