How can we extract invoice fields from messy vendor PDFs into strict JSON so our ERP doesn’t reject records when a field is missing or malformed?

Quick Answer: You get from messy vendor PDFs to ERP-safe JSON by treating extraction as a deterministic pipeline, not a one-off LLM prompt. Route each file, extract into a strict schema, enrich against your own master data, then validate and flag anything that would cause your ERP to reject or mispost before it ever hits the API.

Most invoice projects die in the gap between “the demo worked” and “SAP stopped rejecting half our batch.” The PDFs you get from vendors are chaotic: endless formats, scanned images, handwritten notes, multi-page tables, random attachments. Your ERP is the opposite: rigid schemas, strict enums, and hard validation on dates, currencies, vendor IDs, tax lines, and totals.

That’s the real problem: not “extract an invoice number,” but “produce strict, schema-valid JSON every time, or fail loudly before the ERP does.”

---

Quick Answer: You get from messy vendor PDFs to ERP-safe JSON by treating extraction as a deterministic pipeline, not a one-off LLM prompt. Route each file, extract into a strict schema, enrich against your own master data, then validate and flag anything that would cause your ERP to reject or mispost before it ever hits the API.

Why This Matters

In finance, “mostly accurate” is a bug. A 95% success rate on invoices means daily reconciliations, rejected postings, duplicate payments, and teams living in exception queues. Your ERP doesn’t care that the PDF was skewed or that the vendor changed their layout. It just sees: required field missing, malformed, or inconsistent with its own master data.

If you can reliably turn messy vendor PDFs into strict, schema-valid JSON—with explicit exceptions when something’s off—you stop burning engineering cycles on glue code and finance time on manual fixes. Instead of “AI that tries its best,” you get a pipeline your auditors and controllers can trust.

Key Benefits:

• Fewer ERP rejects: Schema validation and deterministic checks catch missing or malformed fields before they hit SAP, NetSuite, or Dynamics.
• Real automation, not semi-automation: Per-field confidence and exception routing mean your team touches only the edge cases, not every invoice.
• Predictable, debuggable behavior: Versioned functions, evals, and rollback give you production-grade control—no silent regressions when a model changes.

Core Concepts & Key Points

Concept	Definition	Why it's important
Strict JSON schema	A canonical invoice schema (e.g., JSON Schema) that defines types, required fields, enums, and constraints for every field.	Forces consistency across messy PDFs and prevents malformed payloads from reaching the ERP. Schema-valid or flagged—no undefined behavior.
Composable extraction workflow	A pipeline built from primitives like Route, Split, Transform, Enrich, Validate, and Sync that runs per invoice.	Lets you handle mixed packets, multi-page tables, and vendor edge cases with deterministic branching instead of ad-hoc scripts.
Enrichment & validation against master data	Post-extraction checks and lookups against your own vendor list, GL codes, tax rules, and business logic.	Ensures the extracted values are not just syntactically valid but operationally correct for your ERP (e.g., vendor ID exists, tax is plausible, totals reconcile).

How It Works (Step-by-Step)

At bem, we treat “how can we extract invoice fields from messy vendor PDFs into strict JSON so our ERP doesn’t reject records when a field is missing or malformed?” as an end-to-end workflow problem, not a model problem. The flow looks like this:

1. Route & Normalize the Input

   You start by ingesting any input—PDF, image, email, ZIP of mixed documents—via a REST call to a workflow:

   curl -X POST https://api.bem.ai/v1/workflows/invoice-intake-v1/run \
     -H "Authorization: Bearer $BEM_API_KEY" \
     -H "Content-Type: application/json" \
     -d '{
       "inputUrl": "https://files.example.com/vendor-packet-123.pdf",
       "source": "ap_inbox"
     }'
   

   Inside invoice-intake-v1:

   • Route: Detects what’s actually inside the file (Invoice vs Statement vs Credit Memo vs Packing Slip) and routes invoices to the correct workflow.
   • Split: Breaks multi-document packets apart (e.g., one PDF containing a PO, an invoice, and a BOL).
   • Normalize: Unifies images, scans, and text PDFs into a single representation for downstream extraction.

   Mechanism: routing uses layout + content signals, not fragile “file name contains ‘invoice’” rules.

2. Transform: Extract into a Strict Invoice Schema

   Once a file is identified as an invoice, a dedicated transform function extracts fields into a strict JSON schema:

   {
     "functionName": "invoice-extractor-v1",
     "type": "transform",
     "outputSchemaName": "Invoice Schema",
     "outputSchema": {
       "type": "object",
       "required": [
         "invoice_number",
         "invoice_date",
         "currency",
         "vendor_name",
         "line_items",
         "subtotal",
         "tax_total",
         "total_amount"
       ],
       "properties": {
         "invoice_number": { "type": "string" },
         "invoice_date": { "type": "string", "format": "date" },
         "due_date": { "type": ["string", "null"], "format": "date" },
         "currency": { "type": "string", "enum": ["USD","EUR","GBP","CAD","AUD","JPY"] },
         "vendor_name": { "type": "string" },
         "vendor_invoice_reference": { "type": ["string", "null"] },
         "line_items": {
           "type": "array",
           "items": {
             "type": "object",
             "required": ["description","quantity","unit_price","line_total"],
             "properties": {
               "description": { "type": "string" },
               "quantity": { "type": "number" },
               "unit_price": { "type": "number" },
               "line_total": { "type": "number" },
               "gl_code": { "type": ["string", "null"] }
             }
           }
         },
         "subtotal": { "type": "number" },
         "tax_total": { "type": "number" },
         "total_amount": { "type": "number" }
       }
     }
   }
   

   Under the hood:

   • Layout-aware models handle multi-page tables, nested line items, and weird column headers.
   • The output is validated against Invoice Schema in the workflow itself. If the model tries to skip a required field or emit the wrong type, the run is flagged instead of silently passing a broken payload downstream.
   • Every field is annotated with confidence and hallucination checks, so you know where the model guessed vs where it’s solid.

   Result: you get strongly typed JSON per invoice, not “whatever the LLM felt like returning.”

3. Enrich, Validate & Sync to Your ERP

   Extraction alone doesn’t make your ERP happy. It wants its own IDs, codes, and rules. The last step is where you use your data to enforce that.

   Enrich with your Collections

   • Vendor resolution: Map vendor_name like “Amazon Web Services”, “AWS”, or “Amazon.com Services” to a canonical vendor ID in your ERP:

     {
       "vendor_name": "Amazon Web Services",
       "vendor_id": "vnd_9982",
       "vendor_match_confidence": 0.98
     }
     

   • GL code assignment: Hydrate gl_code on line items by matching descriptions to your chart of accounts.

   • Cost center / project mapping: Enrich based on keywords or PO references.

   These lookups run against your Collections—curated lists of vendors, GL codes, cost centers—with match confidence scores. Low-confidence matches can automatically route to a human review surface instead of blindly posting.

   Deterministic validation

   Next, a Validate step runs hard checks on the enriched payload before sync:

   • Required fields present: no null where your ERP requires a value.
   • Types and enums correct: currency codes allowed, tax types valid.
   • Arithmetic consistency: line items sum to subtotal, subtotal + tax = total, within a small tolerance.
   • Business rules: invoice date not in the future or older than your cutoff; total below approval thresholds, etc.
   • Master data existence: vendor_id, GL codes, and tax codes exist in your ERP Collections.

   If any rule fails:

   • The workflow emits a structured exception instead of broken JSON.
   • The invoice is routed to a Surface—a review UI auto-generated from the schema—where an operator can fix fields.
   • Corrections feed back into training data and evals, so the system gets better on real edge cases.

   Sync via ERP-safe payload shaping

   Finally, a Sync step shapes the JSON into your ERP’s ingestion format using JMESPath-style selectors. For example, for NetSuite or SAP, you might map to their invoice creation payload structure. The call is idempotent: re-running the workflow with the same external ID won’t create duplicate invoices.

   Mechanisms that keep your ERP safe:

   • Schema validation before sync.
   • Idempotency keys to prevent duplicate postings.
   • Versioned workflows so you can roll back if a new rule or model version changes behavior.

Common Mistakes to Avoid

• Relying on a single LLM call with a clever prompt:
  A one-shot “extract all invoice fields into JSON” prompt will work in a demo and fail in production. It has no concept of required fields, no schema enforcement, and no feedback loop from your ERP. Use a workflow with strict schemas and evals, not just prompts.

• Treating vendor edge cases as one-off scripts:
  Writing special-case Python for every tricky vendor (“if header says ‘Tax invoice’ then parse differently”) doesn’t scale. You end up with dozens of brittle parsers. Instead, use routing + collections + workflow branches with version control, so you can manage complexity without a pile of ad-hoc code.

Real-World Example

A mid-size logistics company came to us with exactly this problem: tens of thousands of invoices per month, hundreds of vendor formats, and an ERP (SAP) that rejected 20–30% of records due to missing or malformed fields. Their team had already built a Python + OCR pipeline and bolted an LLM on top. It sort of worked—until they went beyond their five “friendly” test vendors.

In production:

• Some invoices had multi-page line-item tables that their per-page OCR couldn’t stitch.
• Vendor names in the PDFs didn’t match SAP’s vendor master data.
• Currency fields were missing or misread, so SAP rejected the payloads.
• When they changed a parsing rule to fix one vendor, others quietly broke.

We implemented a bem workflow:

1. Route & Split: Mixed packets (PO + invoice + packing list in one PDF) were automatically split and routed. Only invoices flowed into the invoice workflow.
2. Transform: invoice-extractor-v1 produced schema-enforced JSON with line items across pages handled automatically.
3. Enrich: Vendor names and line-item descriptions were matched against their SAP vendor list and GL codes via Collections, with match confidence and thresholds.
4. Validate: A deterministic validation step checked totals, currency, and business rules before any SAP call.
5. Sync: A shaped, ERP-safe payload was sent to SAP via their standard API with idempotent external IDs.

After rollout:

• SAP reject rate dropped from ~25% to under 2%.
• The remaining 2% were surfaced as explicit exceptions with per-field confidence, not nasty surprises in SAP.
• Finance stopped hand-typing invoices and focused only on true edge cases.

Pro Tip: Start by defining the exact JSON schema your ERP wants—as it really is in production, not “what would be nice”—then build your extraction workflow around that schema. If a field isn’t in the schema or in your posting rules, don’t extract it “just in case.”

Summary

If you want your ERP to stop rejecting records, you can’t treat invoice extraction as a fuzzy AI problem. You need a deterministic, versioned workflow that:

• Routes and normalizes messy vendor PDFs and packets.
• Extracts fields into a strict, enforced invoice schema with per-field confidence.
• Enriches against your own vendor, GL, and tax Collections.
• Validates against arithmetic and business rules before anything touches the ERP.
• Emits either schema-valid JSON or a structured exception, never silent failures.

That’s how you get from “we have a pile of messy invoices” to “our ERP ingests invoices automatically, and the numbers—including line items—are correct.”

Next Step

Get Started