How should we handle exceptions in document extraction so low-confidence fields go to a review queue and nothing silently writes bad data?

Most document extraction projects don’t fail on accuracy. They fail on silent failures—when a model is “pretty sure” and your system accepts it as truth. If you’re serious about AP, claims, logistics, KYC, or any workflow where money moves, you can’t afford guessing. You need a deterministic exception model: low-confidence fields go to a review queue, and nothing writes to your system of record unless it’s schema-valid and auditable.

Quick Answer: Handle exceptions in document extraction by making confidence, schema validation, and hallucination checks first-class citizens of your pipeline. Every extraction call should either return schema-valid JSON with per-field confidence, or emit a structured exception that routes to a human review queue—never a “best effort” write into your ERP or core app.

Why This Matters

If you’re processing invoices, shipping packets, claims, or onboarding forms at scale, a 95% “accuracy” rate is not a win; it’s a backlog of hidden bugs. One wrong total can trigger a payment error. One mis-routed shipment can cost more than your entire AI budget. The real risk isn’t low-confidence fields—it’s high-confidence mistakes that land in production with no trace.

Exception handling is how you move from “cool demo” to “system your finance team actually trusts.” When you treat extraction like production software—with schema enforcement, confidence thresholds, evals, and explicit review queues—you get three things: predictable behavior, observable failures, and the ability to improve over time without breaking what already works.

Key Benefits:

• No silent bad writes: Schema validation + confidence rules ensure your ERP/claims system only sees trusted, schema-valid JSON (or explicit exceptions).
• Human review where it matters: Low-confidence or rule-violating fields route to review surfaces, so operators focus on the 5–10% of tricky cases, not 100% of documents.
• Continuous accuracy improvement: Exceptions and corrections feed into golden datasets, evals, and regression tests, so your extraction gets measurably better instead of drifting.

Core Concepts & Key Points

Concept	Definition	Why it's important
Schema-enforced extraction	Extraction that must produce JSON conforming to a predefined schema (types, enums, required fields) or emit an exception.	Prevents “best-effort” output; either you get valid data your systems can trust, or an explicit exception to handle.
Per-field confidence & hallucination detection	A score and sanity check for each field indicating how likely it is to be correct and whether the model appears to be inventing data.	Lets you route only uncertain or suspicious fields to review, instead of blindly trusting model output.
Review queues & exception routing	A deterministic flow that sends low-confidence or invalid fields/documents to human review, with full context and audit trails.	Keeps humans in the loop where needed, while preserving throughput, traceability, and regulatory readiness.

How It Works (Step-by-Step)

At Bem, we think about this as a workflow problem, not a “try a better model” problem. A good exception strategy decomposes into clear steps: Route → Extract → Validate → Enrich → Decide → Review or Sync.

Here’s what that looks like concretely.

1. Route and Normalize Inputs

You start with a messy firehose: PDFs, scans, emails, images, ZIPs of shipping packets. The first step is to normalize and identify.

1. Route: A Route function detects document type(s) and splits mixed packets:
   • Example types: Invoice, BillOfLading, CustomsDeclaration, Receipt.
   • Mixed packet? Use Split to separate pages/sections, then Route each part.
2. Normalize: Standardize to an internal representation:
   • Apply OCR if needed.
   • Normalize encodings and page order.
   • Attach metadata (source system, upload user, timestamps).

This is where you also attach an idempotency key so re-runs don’t double-post.

2. Extract with Schema Enforcement

Next, you extract fields into a strict schema, not a free-form blob.

For an invoice, your JSON Schema might include:

{
  "type": "object",
  "required": ["vendor", "invoice_number", "invoice_date", "total", "currency", "line_items"],
  "properties": {
    "vendor": { "type": "string" },
    "invoice_number": { "type": "string" },
    "invoice_date": { "type": "string", "format": "date" },
    "due_date": { "type": "string", "format": "date" },
    "total": { "type": "number" },
    "currency": { "type": "string", "enum": ["USD", "EUR", "GBP", "JPY"] },
    "line_items": {
      "type": "array",
      "items": {
        "type": "object",
        "required": ["description", "quantity", "unit_price"],
        "properties": {
          "description": { "type": "string" },
          "quantity": { "type": "number" },
          "unit_price": { "type": "number" }
        }
      }
    }
  }
}

In Bem, the Extract step:

• Runs extraction against this schema.
• Produces schema-typed fields + a confidence score per field.
• Adds hallucination detection (e.g., if a required field isn’t visible anywhere, we treat guesses as suspect, not “good enough”).

By architecture, you either get:

• Schema-valid JSON with per-field confidence, or
• A flagged exception saying which fields failed schema or confidence checks.

No “95% complete JSON” silently moves forward.

3. Validate, Enrich, and Score Risk

Now you apply business logic and enrichment, which is where most exceptions actually surface.

Typical Validate and Enrich steps:

• Numeric consistency: Check if sum(line_items.amount) == total within a small tolerance.
• Date rules: due_date >= invoice_date, not 10 years in the past.
• Enum checks: category ∈ {AP, AR, PO}, not "random".
• Vendor enrichment: Use a Collection of known vendors to:
  • Match by name, address, tax ID with a match-confidence score.
  • Flag if vendor is unknown or ambiguous.

Each field ends up with:

• Extraction confidence (from the model).
• Validation status (Valid, Flagged, Missing, Inconsistent).
• Enrichment match confidence (e.g., vendor match 0.91 vs 0.62).

Example outcome:

Field	Value	Extract Conf.	Validation	Notes
vendor	ACME INC	0.98	Valid	Matched vendor_id=123 (0.97 match)
total	1250.00	0.99	Valid	Sums to line items within tolerance
due_date	2026-01-12	0.71	Flagged	After allowed terms for this customer
category	“APR”	0.88	Flagged	Not in enum {AP, AR, PO}

Two important design choices here:

• Confidence is not the only signal. A high-confidence but rule-breaking field is just as dangerous as a low-confidence one.
• Validation and enrichment run before persistence. You never persist raw extraction and “hope” downstream services catch it.

4. Decide: Auto-Accept vs Route to Review

Now you implement deterministic decision logic. This is where most teams hard-code if/else in an app server; in Bem, this lives in the workflow as config and functions, with versioning and rollback.

A typical policy:

• Auto-accept a document if:
  • All required fields are present.
  • All required fields have confidence ≥ 0.9.
  • No validation errors on critical rules (totals, dates, enums).
  • Vendor/customer matched above 0.95.
• Flag for review if:
  • Any critical field is below threshold.
  • Any hard validation rule fails (e.g., totals mismatch).
  • A hallucination detector marks a field as suspect (e.g., invented PO number).
  • Enrichment is ambiguous (multiple vendor candidates, low match).

This policy produces one of two outcomes:

1. Status = ACCEPTED

   • Payload is schema-valid and trusted.
   • The workflow calls a Sync function to write to your ERP/claims system via REST, queue, or file drop.

2. Status = EXCEPTION

   • The workflow emits a structured exception event:

     {
       "document_id": "doc_123",
       "status": "EXCEPTION",
       "fields": {
         "due_date": { "value": "2026-01-12", "confidence": 0.71, "reason": "violates_terms" },
         "category": { "value": "APR", "confidence": 0.88, "reason": "enum_violation" }
       }
     }
     

   • That event routes to a review queue (a Bem Surface, your internal UI, or both).

Nothing writes directly to the system of record unless it’s in the “accepted” path.

5. Review Surfaces and Human-in-the-Loop

A review queue is useless if your operators hate using it. The UI needs to be schema-aware, fast, and auditable.

Bem generates Surfaces from your schema:

• Field-level UI: For each exception, it shows:
  • Current value, confidence, and reason for flag.
  • Original document snippet alongside (e.g., cropped area around the field).
  • Enrichment suggestions (e.g., top 3 vendor matches).
• Actions per field:
  • Accept as-is.
  • Correct value (type-safe input: date picker, enum dropdown, currency).
  • Mark as “N/A” or “Not recoverable” if you intentionally leave it blank.
• Actions per document:
  • Approve document and re-run downstream Sync.
  • Reject document with a reason.

Every action is:

• Versioned (who changed what, when).
• Traceable back to the original extraction and model version.
• Replayable via idempotent re-runs.

This design closes the loop: you get clean data and a full audit chain for regulated environments.

6. Feed Corrections Back Into Evals and Models

Exception handling is not just a safety mechanism; it’s your training and eval pipeline.

With Bem, you:

1. Capture corrections as labeled data.
   • For each corrected field: predicted_value, final_value, confidence, document_type, model_version.
2. Build golden datasets.
   • Subset of documents with human-validated truth.
   • Balanced across vendors, layouts, languages, edge cases.
3. Set up automated evals.
   • F1 scores per field: vendor, total, line_items.description, etc.
   • Run on every new version of a function or workflow.
4. Add regression tests.
   • If F1 drops on critical fields, block rollout or auto-rollback.

That’s how you go from 72% → 99% F1 on real workloads without crossing your fingers each time you “improve” the prompt.

In Bem’s world, accuracy isn’t a vibe. It’s a metric with a git history.

Common Mistakes to Avoid

• Treating “mostly correct” as acceptable:
  In AP, claims, or logistics, a 5% error rate is catastrophic. Don’t auto-write any document that fails schema, confidence, or validation checks—no matter how good the demo looks.

• Batching exceptions into generic “error buckets”:
  “Something went wrong” is not observability. Classify exceptions by field, reason, and document type. That’s how you know whether to fix prompts, schemas, enrichment, or upstream scanning.

• Hard-coding business logic in app servers:
  When your validation rules and routing live as scattered if statements across services, you can’t version, test, or rollback safely. Put this logic into workflows/functions with explicit versions and eval gates.

• No idempotency on retries:
  If you re-run a document after fixing a model or schema, you shouldn’t re-post the invoice or re-create the claim. Use idempotency keys and workflow IDs to guarantee safe re-runs.

• Ignoring hallucination detection:
  The worst failures are confident, invented values (like PO numbers or GL codes) that look legit. Use detectors that recognize when the model is guessing beyond the visible evidence and treat these as exceptions, not successes.

Real-World Example

Say you’re processing international shipping packets: Bill of Lading + Customs Declaration + Payment Receipt + Release Order. Everything arrives as a single 4–10 page PDF. Today, your team manually rekeys:

• Shipper/consignee
• Vessel, ports of loading/discharge
• BL numbers, container and seal numbers
• Gross/net weights
• Customs declaration IDs and tax breakdown
• Payment totals and release authorizations

With a naive LLM setup, you prompt “Extract all shipping details” and push the output into your TMS. This works until:

• The BL is low-resolution and the container number is misread.
• The customs declaration format changes and tax amounts swap columns.
• The payment receipt total is correct, but the currency is wrong.

With Bem, the same packet runs through a single workflow:

1. Route & Split:

   • Route identifies the packet type.
   • Split separates BL, Customs, Receipt, Release pages.

2. Extract:

   • Each section is extracted into its schema (BillOfLading, CustomsDeclaration, etc.) with per-field confidence.

3. Validate & Enrich:

   • Container numbers validated against pattern rules.
   • Ports enriched from your port master list with match confidence.
   • Totals validated across customs declaration and receipt.

4. Decide:

   • If all critical fields clear confidence + validation thresholds, the workflow marks the packet ACCEPTED and sends normalized JSON into your TMS.
   • If, say, the customs tax breakdown totals don’t reconcile, or a port is ambiguous, only those specific fields are flagged.

5. Review Surface:

   • Operator sees: original PDF snippet, extracted value, confidence, and suggested port matches.
   • They correct the port and adjust one tax field.
   • On save, the workflow automatically re-runs the validation and posts the clean payload.

6. Learn:

   • Those corrections feed into golden datasets for shipping packets.
   • Evals track F1 on fields like container_number, gross_weight, port_of_discharge.
   • Model/workflow updates roll out only when evals stay green.

The outcome: millions of documents per week, with schema-valid outputs and no silent corrupt writes to your TMS. Exceptions exist, but they’re explicit, routed, and fast to clear.

Pro Tip: When you design your exception model, start with “What absolutely must never be wrong?” (e.g., totals, currencies, account IDs, container numbers) and codify those as hard validation gates with strict confidence thresholds. Everything else can be more permissive, but those critical fields should either be correct or in a review queue—nothing in between.

Summary

Exception handling in document extraction is not a “nice to have.” It’s the difference between a demo and a system that finance, ops, and compliance can sign off on.

The pattern that works:

• Define strict schemas for each document type.
• Attach per-field confidence and hallucination detection to every extraction.
• Run validation and enrichment before you even think about persistence.
• Use deterministic workflows to route:
  • Clean, schema-valid outputs to your systems of record.
  • Low-confidence or rule-breaking fields to a review queue with a schema-aware UI.
• Feed exceptions and corrections into evals, golden datasets, and regression tests so accuracy improves with every document instead of drifting.

Agents guess. Ad-hoc wrappers break. A production-grade pipeline doesn’t. It either returns data you can trust—or an explicit exception that your team can clear.

Next Step

Get Started