Why do LLM extraction pipelines sometimes “make up” totals or dates, and how do we design a system that fails closed instead of guessing?

Quick Answer: LLM extraction pipelines “make up” totals or dates because they’re sampling engines trained to always produce something, even when the input is ambiguous, low quality, or missing. To design a system that fails closed instead of guessing, you have to wrap the LLM in a deterministic pipeline: strict schemas, per-field confidence, cross-checks against business rules, and exception routing when the model isn’t sure.

Why This Matters

If you’re processing invoices, claims, packets, or contracts, “mostly accurate” is a bug. A hallucinated invoice total isn’t a UX issue; it’s a payment error, a compliance risk, or a reconciled ledger that doesn’t actually reconcile. The gap between “we extracted your data” and “your operation is complete” is where every proof of concept dies. Designing for fail-closed behavior is how you turn stochastic LLM output into something your ERP, GL, and regulators can trust.

Key Benefits:

• Fewer silent failures: Bad or missing data gets flagged and routed, not smuggled into your system as “best effort” JSON.
• Deterministic behavior in production: The same input and workflow version produce the same result, or a clear exception, every time.
• Auditable accuracy over time: You can measure, debug, and improve extraction with golden datasets, evals, and regression tests instead of praying the prompt still works.

Core Concepts & Key Points

Concept	Definition	Why it's important
Stochastic LLM behavior	LLMs generate outputs by sampling from probability distributions over tokens, not by “knowing” a correct answer.	Explains why they confidently fabricate totals/dates instead of returning “unknown” when data is unclear or missing.
Fail-open vs fail-closed	Fail-open: pipeline returns “best guess” data even when uncertain. Fail-closed: pipeline refuses to commit and emits an explicit exception.	Determines whether your system silently corrupts downstream operations or forces a safe review path.
Schema-enforced workflows	Pipelines that enforce typed JSON schemas, cross-field constraints, and business rules around LLM calls.	Converts probabilistic extraction into deterministic, auditable behavior and gives you hooks for evals, drift detection, and human review.

Why LLM Extraction Pipelines “Make Up” Totals or Dates

Let’s start with the uncomfortable truth: the model is doing exactly what it was trained to do.

1. LLMs are trained to complete patterns, not to be right

The model sees: “Here is an invoice. Extract the total and invoice_date as JSON.”

Even if:

• The PDF is low-res or partially cropped
• The OCR dropped a line
• The “Total” is illegible or spread across two pages
• The date appears in ambiguous formats like 03/07/24

…the LLM’s job is: produce a plausible-looking completion. It doesn’t know that your AP team would rather get an exception than a wrong number.

Patterns it happily leans on:

• “Totals” on invoices often look like 1234.56
• Dates often look like 2024-03-07 or 03/07/2024
• If there’s a subtotal + tax but no visible total, it might just add them and call it “total_amount”

From the LLM’s perspective, this is a success. You got structured JSON. From a production perspective, you just quietly overpaid or misbooked a liability.

2. Naive prompts turn “I don’t know” into “just pick something”

Most extraction prompts are imperative and unsafe:

“Extract the invoice_number, invoice_date, due_date, and total_amount. Always fill all fields. If something is missing, infer it from context.”

This is a recipe for hallucination:

• “Always fill all fields” is a direct instruction to never fail.
• “Infer from context” gives permission to manufacture totals/dates that “look right.”

Without explicit instructions and downstream validation that refuses half-baked values, the model will happily make up:

• Due dates based on typical Net-30 patterns
• Invoice numbers based on nearby alphanumeric strings
• Totals that roughly match a guessed sum of line items

3. OCR and layout noise amplify hallucinations

In real pipelines, the LLM rarely sees the original PDF. It sees the OCR output and some layout hints. Common failure modes:

• OCR dropped the bottom of the page where the total lives
• A multi-page table is split incorrectly, so line items are incomplete
• Currency symbols get detached or misread ($ → S, € → C)

Per-page tools are blind to packet-level context:

• They see one page with line items but no total, and they “helpfully” guess.
• They see a stray date in a footer and treat it as invoice_date.

If you don’t cross-check fields against each other and against your own rules (e.g., “total must equal sum(line_items) ± allowed rounding”), you never notice the disconnect.

4. Most systems are architected to fail open

Even if the model is uncertain, your pipeline often:

• Accepts any JSON that parses
• Defaults unknowns to null or a fallback date
• Skips confidence thresholds or validation checks

Result: the database ends up with schema-valid, semantically wrong data. It looks clean. It behaves like a time bomb.

A fail-closed system inverts this:

• Either you get schema-valid, cross-checked data
• Or you get a structured exception with a reason and a review path

No in-between “vibes-OK” state.

How It Works (Step-by-Step)

Here’s how we design pipelines at Bem so they fail closed instead of guessing. Stochastic LLM calls inside; deterministic behavior outside.

1. Define strict schemas and invariants up front
   You start by defining a JSON Schema for each business object, not a vague list of fields in a prompt.

   Example: Invoice schema snippet:

   {
     "type": "object",
     "required": ["invoice_number", "invoice_date", "currency", "total_amount", "line_items"],
     "properties": {
       "invoice_number": { "type": "string", "minLength": 1 },
       "invoice_date": { "type": "string", "format": "date" },
       "currency": { "type": "string", "enum": ["USD", "EUR", "GBP", "MXN"] },
       "total_amount": { "type": "number", "minimum": 0 },
       "line_items": {
         "type": "array",
         "items": {
           "type": "object",
           "required": ["description", "quantity", "unit_price", "line_total"],
           "properties": {
             "quantity": { "type": "number", "minimum": 0 },
             "unit_price": { "type": "number", "minimum": 0 },
             "line_total": { "type": "number", "minimum": 0 }
           }
         },
         "minItems": 1
       }
     }
   }
   

   Then codify invariants as rules:

   • abs(sum(line_items.line_total) - total_amount) <= 0.01
   • due_date >= invoice_date
   • invoice_date <= today + 1 day

2. Wrap LLM extraction in a schema-enforced function
   Inside Bem, this becomes a Transform function: extract_invoice_v3.

   Characteristics:

   • Input: file blob or pre-processed text/layout + your JSON Schema
   • Output: candidate JSON + per-field confidence + hallucination signals
   • Behavior: the function must either:
     • Return schema-valid JSON with confidence scores, or
     • Raise an exception (e.g., “required fields missing” or “format violations”)

   We use:

   • Temperature/decoding tuned for extraction (low variance, less “creative”)
   • Multiple passes or constrained decoding to reduce structural errors
   • Internal checks for “model said something exists but we can’t locate it in text” → hallucination detection

3. Add workflow-level validation, routing, and review

   A production workflow isn’t just “call the model.” It’s a pipeline:

   1. Route: detect document type (invoice vs credit memo vs packing slip). Wrong type? Exception.
   2. Transform: call extract_invoice_v3 with your schema.
   3. Validate: apply business rules and invariants:
      • Check totals vs line items
      • Validate dates and currency codes
      • Enforce vendor-specific rules (e.g., known tax rates)
   4. Enrich: match vendor name against your Vendor Collection with match confidence; flag mismatches.
   5. Decide: based on per-field confidence and validation results:
      • If all required fields pass invariants and confidence ≥ threshold → mark as ready_for_sync
      • Else → mark as needs_review and attach failure reasons
   6. Surface: send needs_review cases to a human review UI (a “Surface”) auto-generated from the schema.
   7. Sync: only after approval, push to your ERP/AP system via REST.

   Behavior is now binary:

   • You get clean, schema-valid, business-rule-compliant JSON
   • Or you get a clear, traceable exception with all intermediate artifacts

   No “the model tried its best, so we shipped it anyway.”

Common Mistakes to Avoid

• Treating prompts as contracts instead of hints
  If you rely solely on “be accurate, do not guess” in the prompt, you’re outsourcing the contract to a stochastic system. Wrap the model in code-level contracts: JSON Schema, invariants, confidence thresholds, and review queues.

• Skipping per-field confidence and cross-checks
  Many teams log a single “model confidence” or nothing at all. Instead:

  • Capture confidence per field (invoice_date, total_amount, etc.)
  • Cross-validate related fields (sum of line items vs total, date consistency)
  • Route borderline or low-confidence fields to human review, not straight to production

Real-World Example

A customer came to us after trying a standard “document AI” provider for invoices. Demo looked great. In production, they discovered:

• Some invoices had missing totals due to bad scans
• The system “helpfully” summed visible line items and wrote that as total_amount
• Tax lines sometimes got double-counted or dropped

AP only noticed when vendor statements didn’t match ledger totals. By then, hundreds of invoices had been pushed downstream.

We rebuilt the pipeline on Bem:

1. Route each packet to the right workflow (invoice, credit memo, PO, etc.).
2. Transform invoices through a versioned extract_invoice_v4 function with:
   • Strict schema
   • Per-field confidence
   • Hallucination detection when claimed fields don’t exist in the underlying text
3. Validate:
   • sum(line_total) vs total_amount within tolerance
   • tax_amount and tax_rate consistent with their internal tax rules
   • invoice_date not more than 90 days in the future or 10 years in the past
4. Enrich vendor names against their Vendor Collection with match confidence.
5. Route failures (low confidence, invariant failures) to a review Surface:
   • AP operators see the original PDF, extracted JSON, and exact validation errors
   • Fixes are captured as structured data and fed back into evals
6. Sync only validated+approved invoices to their ERP.

Outcome:

• Totals including line items are now correct, or the invoice is explicitly in an exception queue.
• They track F1 scores and pass rates across vendors, versions, and time.
• When they change the workflow, they run regression tests before promoting a new version.

No more “the model made something up and we didn’t notice.”

Pro Tip: In your eval suite, explicitly include “missing total” and “ambiguous date” cases. Your bar isn’t “the model guesses right”; your bar is “the workflow refuses to guess and clearly flags the edge case.”

Summary

LLM extraction pipelines “make up” totals or dates because they’re doing probability, not truth. If you tell a sampling engine to always produce full JSON and you don’t enforce any contracts around it, it will happily invent plausible numbers and dates that pass casual inspection and quietly corrupt your operation.

A production-ready system inverts the responsibility:

• The model proposes candidates.
• Your workflow enforces schemas, invariants, confidence thresholds, and business rules.
• When something doesn’t add up, the system fails closed with an explicit exception and a human review path.

At Bem, we treat accuracy like software quality: versioned functions and workflows, golden datasets, evals, regression tests, and rollback. Trainable. Deterministic at the system level. It never guesses silently.

Next Step

Get Started