Embeddings & Reranking Models
ZeroEntropy vs BGE-M3 / Qwen embeddings: should we pay for an API or self-host open models for our workload?
7 min read
Quick Answer: If you care about predictable latency, higher NDCG@10, and not running an infra Frankenstein, use a purpose-built embedding API like ZeroEntropy for retrieval-heavy workloads and self-host open models (BGE-M3, Qwen, zembed-1 weights) only when data sovereignty and tight infra control are non‑negotiable.
Frequently Asked Questions
When does it make sense to pay for an embedding API instead of self-hosting BGE-M3 or Qwen?
Short Answer: Pay for an embedding API when retrieval quality, latency SLAs, and operational simplicity matter more than bare metal cost; self-host when you’re blocked by data residency, VPC-only policies, or have a strong infra team ready to own GPUs and scaling.
Expanded Explanation:
BGE-M3 and Qwen embeddings are strong open models, and for many teams they look “free” compared to paying per million tokens. The catch is that quality, throughput, p99 latency, observability, and maintenance are now entirely your problem. You’re suddenly in the business of running model infra, not just building retrieval systems.
ZeroEntropy’s zembed-1 is designed as a retrieval-first embedding model with state-of-the-art accuracy and sub-200ms API latency at $0.05/M tokens, plus the option to download the same weights and self-host when you truly need to. For most RAG and search applications, the fully managed API gives you better NDCG@10, better latency behavior, and lower total cost of ownership than rolling your own BGE/Qwen stack.
Key Takeaways:
- Use a managed API like ZeroEntropy when you want strong retrieval quality, predictable latency, and minimal ops.
- Reach for self-hosted BGE-M3/Qwen/zembed-1 when regulatory constraints or strict VPC-only policies rule out third‑party APIs.
How should we evaluate ZeroEntropy vs BGE-M3 / Qwen for our specific workload?
Short Answer: Treat this as a retrieval benchmark problem: compare NDCG@10, recall@k, and p50/p99 latency for your own corpus and query mix, and include operational overhead and GPU cost in the calculation—not just token price.
Expanded Explanation:
The right way to choose between ZeroEntropy and self-hosted BGE-M3/Qwen is to run a focused evaluation on your data. That means: sample real user queries (not synthetic prompts), label relevance for top‑k results, and measure ranking quality (NDCG@10, recall@50) plus end‑to‑end latency. After that, translate infra into real dollars: GPU hours, on‑call, maintenance, and integration complexity.
With ZeroEntropy, you can hit zembed-1 via the embedding API or the unified Search API and measure quality in a few lines of code. With BGE-M3 or Qwen, you’ll need to stand up inference (or use a third‑party host), wire in a vector store, and potentially add your own reranker to reach comparable quality. The evaluation will usually surface that “cheap” self-hosting isn’t cheap once you include the stack around the model.
Steps:
- Define your benchmark set: 200–1,000 real queries with labeled relevant documents/snippets.
- Run side‑by‑side retrieval:
- ZeroEntropy: zembed-1 via API (or Search API)
- Self-hosted: BGE-M3/Qwen embeddings + your vector DB (optionally with your reranker)
- Compare metrics and cost: NDCG@10, recall@k, p50/p99 latency, GPU hours, and estimated LLM token spend driven by each approach.
How do ZeroEntropy embeddings compare to BGE-M3 and Qwen on quality, latency, and cost?
Short Answer: zembed-1 is optimized for retrieval with high accuracy, sub‑200ms latency, and $0.05/M tokens; BGE-M3 and Qwen can be competitive on quality but you trade off predictable latency and total system cost unless you invest heavily in infra.
Expanded Explanation:
BGE-M3 and Qwen embeddings are strong open-weight baselines for hybrid retrieval, but they’re generic models that you still have to operationalize. You’re on the hook for GPU sizing, autoscaling policies, cold starts, and monitoring. Latency, especially p95/p99, is often where self-hosted stacks fall apart under real traffic.
ZeroEntropy’s zembed-1 is built specifically for “human-level” retrieval workloads—RAG, agent tools, enterprise search—and priced at $0.05 per million tokens, which is significantly cheaper than comparable high-quality closed models (OpenAI text-embedding-3-large at ~$0.13/M, Cohere embed-v4.0 at ~$0.12/M). You get state-of-the-art retrieval performance, calibrated behavior across large corpora, and latency engineered for production. And if you still want self-host, you can download the zembed-1 weights and run them in your own VPC, just like BGE-M3.
Comparison Snapshot:
- Option A: ZeroEntropy (zembed-1 via API)
Retrieval-tuned accuracy, sub‑200ms p50 latency, $0.05/M tokens, no GPU ops, SOC 2 Type II / HIPAA options, EU endpoint. - Option B: Self-hosted BGE-M3 / Qwen / zembed-1
Strong base models, full infra control, but you own GPU cost, scaling, monitoring, and p99 behavior. - Best for:
- ZeroEntropy API: teams shipping production RAG/agents/search that want better NDCG@10 and predictable latency without infra overhead.
- Self-host: teams with hard data residency restrictions or existing GPU platform that must keep all text inside their VPC.
How would we actually implement ZeroEntropy vs a self-hosted BGE-M3/Qwen stack?
Short Answer: With ZeroEntropy you provision an API key, call the embeddings or Search API from your app, and skip infra; with BGE-M3/Qwen you stand up inference, wire a vector DB, tune hybrid retrieval, and likely add your own reranker on top.
Expanded Explanation:
The main implementation difference is who owns the retrieval stack. With ZeroEntropy, you call zembed-1 (or the Search API) directly from your backend in a few lines of code; hybrid dense+sparse retrieval and reranking are handled on our side. You don’t touch GPUs, autoscaling, or model updates, and you can still choose EU-region endpoints or go full on‑prem/VPC via ze-onprem if needed.
With self-hosted BGE-M3/Qwen, you’ll either manage your own GPU cluster or pay a hosting provider. You then integrate a vector database, pick your index type, configure hybrid BM25+vector retrieval, and often add a reranker model to fix ranking quality. Each piece adds latency, cost, and operational complexity. It’s workable if you already have a strong MLOps team, but it’s rarely the fastest path to shipping.
What You Need:
- For ZeroEntropy (managed API path):
- API key and SDK call (embeddings or Search API).
- Optional: EU endpoint (eu-api.zeroentropy.dev) or ze-onprem for VPC/on‑prem.
- For Self-hosted BGE-M3 / Qwen / zembed-1:
- GPU infrastructure or hosted inference provider.
- Vector DB + hybrid retrieval config + optional reranker service.
Strategically, how should we decide between paying for ZeroEntropy and self-hosting open embeddings?
Short Answer: Anchor the decision on total retrieval system ROI: quality (NDCG@10), reliability (p99 latency and uptime), and end‑to‑end cost (LLM tokens + infra + ops), not just “API vs free open weights.”
Expanded Explanation:
Embedding choice is no longer about who has the largest dimension size; it dictates whether your RAG and agents feel “human-level” or brittle. If your retrieval stack is weak, your LLM spends more tokens reconstructing missing context, hallucination rates climb, and engineers lose time debugging ranking failures instead of shipping features.
ZeroEntropy’s strategy is simple: combine zembed-1, hybrid dense+sparse retrieval, and calibrated rerankers (zerank-2, trained with our zELO scoring system) in one stack so you don’t have to juggle BM25 weights, thresholds, and multiple providers. You get better top‑k precision, more predictable p50–p99 latencies, and lower LLM token spend because you send fewer, higher-quality chunks into generation. Self-hosting BGE-M3 or Qwen can make sense if you’re optimizing for sovereignty above everything else, but you’ll be rebuilding a lot of retrieval machinery we already ship in a single API.
Why It Matters:
- Impact on reliability: Better retrieval (higher NDCG@10, calibrated scores) directly cuts hallucinations and “lost-in-the-middle” failures in RAG and agents.
- Impact on cost: Stronger embeddings and reranking let you shrink context windows and reduce LLM calls, often dwarfing any per‑million‑token savings from self-hosting.
Quick Recap
Choosing between ZeroEntropy and self-hosted BGE-M3/Qwen isn’t an ideology question; it’s a retrieval systems question. For most production workloads, a managed embedding and search stack like ZeroEntropy—zembed-1 plus hybrid retrieval and rerankers—will deliver higher NDCG@10, tighter p99 latency, and lower overall RAG spend than DIY hosting. When governance or sovereignty demands it, you can still self-host open-weight models, including zembed-1, entirely inside your VPC or on-prem. The right path is the one that makes retrieval quality measurable, reliable, and cheap enough to scale.