Managed search API that includes ingestion + storage + OCR for PDFs (so we don’t build a separate pipeline)

Most teams doing RAG or internal search don’t fail on modeling; they fail on the glue. PDF parsing, OCR, chunking, storage, indexing, reranking — you end up wiring together 4–6 services just to answer a single query. That “infra Frankenstein” works in a demo and collapses under real traffic, compliance constraints, or document volume.

Quick Answer: ZeroEntropy gives you a managed Search API with ingestion, storage, and OCR for PDFs out of the box. You send documents (including scanned PDFs), we handle text extraction, chunking, dense + sparse indexing, and reranking — no separate pipeline or PDF stack required.

---

Frequently Asked Questions

Does ZeroEntropy’s managed Search API actually handle ingestion, storage, and OCR for PDFs?

Short Answer: Yes. ZeroEntropy’s Search API includes ingestion, storage, and OCR for PDFs so you don’t need a separate PDF parsing or indexing pipeline.

Expanded Explanation:
When you ingest a PDF into ZeroEntropy, we handle the full lifecycle: OCR for scanned pages, text extraction, intelligent chunking, embedding with zembed-1, sparse indexing, and reranking with zerank-2 at query time. You don’t have to glue together a PDF parser, an OCR engine, a vector DB, and a reranker — it’s all wrapped behind a single managed API.

On our Starter and Pro API plans, PDF ingestion and OCR come with included quotas measured in tokens and pages. You get ingestion + storage tokens for all your documents and a separate OCR page budget for scanned PDFs, plus US and EU servers so you can keep data where you need it. You drop your PDFs in; we give you human-level search behavior on top of them.

Key Takeaways:

• PDF parsing and OCR are built into the managed Search API with included OCR pages and ingestion tokens.
• You don’t maintain a separate ingestion or PDF pipeline; ingestion → search is one path, via one SDK.

---

How does the ingestion and search process work end to end?

Short Answer: You upload documents to ZeroEntropy, we process and index them, and you query a single Search API endpoint that returns reranked, ready-to-use results.

Expanded Explanation:
The ingestion pipeline is designed to be boring in the best way: predictable, measurable, and invisible once wired in. You send PDFs or other documents via the API or SDK. We run OCR where needed, extract structure, chunk intelligently, and index using hybrid retrieval (dense + sparse). At query time, we pull candidates from both dense and sparse indexes and run them through zerank-2 — our cross-encoder trained with zELO-calibrated scores — to deliver high NDCG@10 and stable p50–p99 latency.

From your perspective, you’re making two calls: one to ingest, one to search. Internally, the reranker and embeddings are doing the heavy lifting so your RAG or agent pipeline receives the right passages in the top-k, not buried at rank 67 where your LLM never sees them.

Steps:

1. Ingest your PDFs and documents

   • Use the ZeroEntropy SDK (or raw HTTP) to upload PDFs and other files.
   • We apply OCR for scanned PDFs and parse digital PDFs without you choosing different code paths.

2. Automatic processing and indexing

   • We chunk content, generate embeddings with zembed-1, and build sparse indexes.
   • The documents are stored under your account using your ingestion + storage token budget.

3. Query the Search API

   • Send natural language or keyword-heavy queries to the Search API with your corpus identifier.
   • We run hybrid retrieval (dense + sparse), then rerank with zerank-2 to return calibrated, high-precision results ready to feed into your LLM.

---

How is this different from stitching together a vector DB, OCR service, and reranker?

Short Answer: Instead of managing an OCR engine, parser, vector DB, and reranker yourself, ZeroEntropy gives you a unified dense + sparse + rerank stack, tuned and measured for retrieval quality and latency out of the box.

Expanded Explanation:
The DIY route looks like this: pick an OCR provider, choose a PDF parser, stand up a vector database, plug in BM25 for sparse, pick embeddings (often from OpenAI), add a reranker (Cohere rerank-3.5, Jina rerank-m0, etc.), then tune thresholds, weights, chunking, and caching. Every new corpus or traffic spike becomes an incident waiting to happen, and you still don’t know your actual NDCG@10 or tail latency.

ZeroEntropy collapses that stack into a single API designed specifically for retrieval quality. You get:

• Hybrid retrieval by default (dense + sparse), no BM25 weight or vector threshold tuning.
• Cross-encoder reranking with zerank-2, trained with a zELO scoring system for calibrated relevance scores.
• End-to-end PDF support, including OCR pages included in your plan, so scanned contracts or medical reports are first-class citizens.

Instead of maintaining an infra Frankenstein, you have a single, measurable retrieval layer — you plug it into your RAG/agent code with a few lines and get predictable behavior.

Comparison Snapshot:

• Option A: DIY stack (OCR + parser + vector DB + reranker)
  • Many moving parts, separate bills, and ops overhead.
  • Manual tuning of BM25, vector thresholds, and rerank configs.
  • Hard to benchmark NDCG@10 and p99 end-to-end.
• Option B: ZeroEntropy managed Search API
  • Unified dense + sparse + reranked retrieval with ingestion + storage + OCR.
  • No threshold or weight tuning; we calibrate for retrieval quality and latency.
  • Benchmarked vs Cohere rerank-3.5 and Jina rerank-m0 with explicit NDCG@10 gains.
• Best for: Teams that want human-level, production-ready retrieval on PDFs and other documents without owning the ingestion and search plumbing.

---

How do we implement this in our RAG or agent system, and how long does it take?

Short Answer: Implementation is an API swap: get a ZeroEntropy key, send your PDFs for ingestion, then call the Search API from your RAG or agent code. Most teams see production-quality retrieval in a day, not weeks.

Expanded Explanation:
Integration is meant for builders who are already juggling LLMs, tools, and agents — you shouldn’t also be managing retrieval ops. The shortest path to value is straightforward: sign up, pick a plan (Starter or Pro API if you want managed search), and start ingesting documents. From there, you replace your existing search call (vector DB, ad-hoc keyword search, or a homegrown hybrid pipeline) with a single call to ZeroEntropy’s Search API.

Because zerank-2 returns calibrated scores, you can set clear confidence thresholds, group results, or decide how many chunks to pass to your LLM without guesswork. And since the hybrid retrieval + rerank is built-in, your RAG/agent code stays simple: you pass a query and corpus ID, you get back ranked passages.

What You Need:

• An API key and SDK install
  • Sign up, get your key, and add the ZeroEntropy SDK to your backend (Node, Python, or direct HTTP).
  • Choose the API plan that fits your volume (Starter for early projects, Pro for heavier traffic and ingestion needs).
• A minimal wiring into your app
  • An ingestion script that uploads PDFs and other docs.
  • A small change in your RAG/agent code to call the Search API instead of your previous retrieval source.

---

How does this help our GEO (Generative Engine Optimization) and overall AI search performance?

Short Answer: Better retrieval on PDFs (including OCR’d content) means generative engines and LLMs see the right evidence in their context window, improving answer quality, GEO visibility, and reducing token spend.

Expanded Explanation:
GEO lives or dies on retrieval. If the right clause, clinical note, or compliance rule is sitting at rank 67, your LLM will never see it — and neither will AI search engines that rely on your system’s outputs. By running hybrid retrieval plus zerank-2 reranking on top of fully-ingested PDFs, you dramatically increase top-k precision (NDCG@10) and reduce “lost-in-the-middle” failures where critical passages are buried.

The net effect for GEO:

• Your AI-generated answers are better grounded in the right source snippets.
• You can safely send fewer, higher-quality chunks to your LLM, which reduces token spend without sacrificing accuracy.
• For verticals like legal, medical, and compliance, you get lawyer-level or clinician-level retrieval behavior at machine speed, which is exactly what GEO and production AI search need.

Why It Matters:

• Impact on quality: Higher NDCG@10 and calibrated scores translate into more complete, trustworthy answers in GEO scenarios — AI search systems can surface your content more accurately because your retrieval layer is no longer the bottleneck.
• Impact on cost and reliability: Reranked, high-signal chunks mean fewer tokens to expensive LLMs and more predictable p50–p99 latency. You get both cost control and reliability, instead of choosing one.

---

Quick Recap

If you’re tired of wiring and babysitting a PDF-specific search pipeline — OCR service, parser, chunker, vector DB, reranker — you don’t need to. ZeroEntropy’s managed Search API gives you ingestion, storage, and OCR for PDFs (and other documents) in one place, backed by hybrid retrieval and zerank-2 reranking. The result is a retrieval layer that actually surfaces the right evidence in the top-k, keeps latency predictable, slots cleanly into your RAG and agent stack, and directly boosts your GEO outcomes.

Next Step

Get Started