How do teams regression test document extraction so a model/prompt change doesn’t break production accuracy overnight?

Model or prompt changes shouldn’t be a leap of faith. If document extraction is feeding AP, claims, onboarding, or your core product, you need the same discipline you apply to code: tests, coverage, and a hard “do not ship” line when accuracy regresses.

Quick Answer: Teams regression test document extraction by maintaining golden datasets, defining strict schemas and field-level metrics, and automatically re-running historical payloads against every new model or prompt version. They gate promotion to production on F1/precision/recall thresholds, compare versioned results, and route any low-confidence or breaking changes into human review instead of silently shipping regressions.

Why This Matters

Most teams get burned the same way: a “small” prompt tweak or model upgrade ships, demos look better, then a week later finance is chasing down missing totals or ops finds that one carrier’s packet stopped parsing. Without regression testing, you’re flying blind. You can’t tell if your extraction is drifting, if a new layout broke existing logic, or if a model update quietly tanked recall on a critical field.

Regression testing for document extraction turns stochastic behavior into enforceable guarantees. You know, empirically, whether the new version is safer, riskier, or just different. And you make changes with audit trails, not vibes.

Key Benefits:

• No silent regressions: Catch when a new model/prompt breaks edge cases or key vendors before it hits production.
• Measurable accuracy: Track precision, recall, and F1 scores per field and per document type instead of hand-waving “it seems better.”
• Safe iteration velocity: Ship changes faster because you can roll out, test against golden datasets, and rollback with confidence when something fails.

Core Concepts & Key Points

Concept	Definition	Why it's important
Golden Datasets	A curated set of real documents with ground-truth labeled outputs used as the source of truth for evaluation.	Gives you a stable baseline to detect regressions when prompts/models/workflows change.
Field-Level Metrics (Precision, Recall, F1)	Quantitative measures of how accurately each field is extracted and validated against ground truth.	Lets you see if a change improved totals but hurt line items, or fixed dates but broke vendor IDs.
Versioned Functions & Workflows	Every extraction function/prompt and workflow is tracked as a versioned artifact with rollback.	Makes it possible to compare before/after behavior and safely revert when a regression is detected.

How It Works (Step-by-Step)

The core process is simple: lock in a ground truth, re-run it on every change, and block promotion when accuracy drifts. On Bem, this is built into the platform, but the principles apply even if you’re gluing this together yourself.

1. Define the contract (schema + fields that matter):
   Start by describing what “correct” looks like in code, not in prose.

   • Define a JSON Schema that mirrors your downstream system:
     • Required fields (e.g., invoice_number, invoice_date, total_amount)
     • Types (string, number, date), enums (currency codes), and formats
     • Nested structures (line items, coverage sections, inspection findings)
   • Identify business-critical fields:
     • Totals, taxes, and line items in AP
     • Policy IDs, claim numbers in insurance
     • VIN, container ID, route codes in logistics
   • Decide where you’ll enforce schema-valid or exception. No gray zone.

   This schema becomes the lens for all your tests. If a new model can’t produce schema-valid JSON, it doesn’t ship.

2. Build golden datasets from real documents:
   Synthetic PDFs don’t break you; real-world packets do.

   • Pull a representative sample from production:
     • Different vendors, layouts, languages, and file types
     • Old scans, skewed pages, handwritten notes, stamped corrections
     • Edge cases: missing fields, credits vs invoices, multi-doc packets
   • Label them once with ground truth JSON:
     • Use a labeling UI (Bem’s Surfaces or your own) so your operators can:
       • Correct extracted values
       • Normalize to your schema (dates, currency, codes)
       • Mark intentionally empty fields (e.g., no tax)
   • Tag datasets by scenario:
     • ap_invoices_multivendor_v1
     • carrier_packets_us_only_q1_2025
     • claims_auto_handwritten_notes

   On Bem, these golden datasets become inputs to Automated Evals, which run statistical analysis on every model or workflow update.

3. Version and instrument every change:
   Never edit a production prompt or model in-place. Treat it like code.

   • Version your extraction logic:
     • functions.extract_invoice_v3
     • workflows.process_claim_packet_v5
   • Capture:
     • Model version (e.g., gpt-4.5-2025-03-01)
     • Prompt template and parameters
     • Routing logic (e.g., confidence thresholds, OCR vs LLM path)
   • Keep an auditable history:
     • Who changed what, when, and why
     • Associated eval runs and results
     • Links to relevant golden datasets

   On Bem, functions and workflows are first-class versioned objects with rollback, so you can switch back when a regression is detected in production.

4. Re-run historical payloads on the new version (regression test):
   This is the core of regression testing: same inputs, new logic.

   • For each golden dataset:
     • Re-run all documents through the new function/workflow version.
     • Compare outputs to ground truth at the field level.
   • Compute metrics:
     • Precision (how many extracted values are correct vs false positives)
     • Recall (how many ground truth values were captured vs missed)
     • F1 score (harmonic mean of precision and recall)
   • Slice results:
     • By field (e.g., total_amount, line_items.description)
     • By document type/vendor
     • By input quality (good scan vs low-res fax)

   On Bem, this shows up as Automated Evals with F1 scores and pass rates like “98.7% pass rate, last 24 hours +1.2%.”

5. Define hard gates and acceptable drift:
   You need rules, not gut feelings.

   Examples:

   • Global gates:
     • “Do not promote if overall F1 drops by >0.3 points on any golden dataset.”
     • “Block if pass rate falls below 98% on ap_invoices_multivendor_v1.”
   • Per-field gates:
     • total_amount: F1 must be ≥ 0.995
     • tax_amount: F1 must be ≥ 0.98
     • line_items.quantity: F1 must be ≥ 0.97
   • Vendor-specific gates:
     • “Carrier X invoices must maintain ≥ 99% pass rate.”

   On Bem, you can enforce “schema-valid output or exception” plus confidence thresholds, so low-confidence fields never sneak into production as “maybe correct.”

6. Route low-confidence and breaking cases to humans, not production:
   When a change introduces uncertainty, treat it as an exception, not a guess.

   • Set per-field confidence thresholds:
     • “If we aren’t 99% sure, route total_amount to human review.”
   • Use a human-in-the-loop queue:
     • Operators review flagged cases in a UI generated from your schema.
     • Corrections:
       • Update ground truth.
       • Feed back to your extraction function as fine-tuning data.
   • Close the loop:
     • Re-run regression tests after fine-tuning/model updates.
     • Confirm that fixes improved both new and legacy cases.

   Bem bakes this in: low-confidence fields are routed to Surfaces, corrections auto-train the function, and regression tests ensure you didn’t break old logic.

7. Monitor drift continuously, not just at release time:
   Regression testing isn’t a one-time pre-release event. Models drift. Inputs change.

   • Run scheduled evals:
     • Nightly or weekly regression runs on golden datasets.
     • Alert if F1 drops beyond a threshold or pass rate trends down.
   • Track live test results:
     • “98.4% passing in last 24 hours, +1.2% vs prior period.”
   • Promote with intent:
     • Canary rollout of a new workflow version to a subset of traffic.
     • Compare live metrics between versions before full rollout.

   On Bem, regression testing and drift detection are first-class, so you see the impact of a change across time, not just at merge.

Common Mistakes to Avoid

• Treating “demo accuracy” as production readiness:
  How to avoid it: Never ship a model/prompt based on a handful of hand-picked PDFs. Require it to pass against your golden datasets with defined thresholds before promotion.

• Evaluating at the document level instead of field level:
  How to avoid it: Measure precision/recall/F1 per critical field. A doc can “look right” while totals or IDs are wrong. Your ERP doesn’t care if the logo was extracted; it cares if the money and identifiers are correct.

• Ignoring schema validation and letting partial/invalid JSON through:
  How to avoid it: Enforce strict schema validation and treat failures as explicit exceptions. Don’t let malformed payloads or missing required fields sneak into downstream systems.

• Not versioning prompts, functions, and workflows:
  How to avoid it: Treat prompts like code. Version them, store them, and tie them to eval results. Make rollback a one-line change, not a forensic exercise.

• No human-in-the-loop path for edge cases:
  How to avoid it: Route low-confidence or unknown patterns into a review queue instead of forcing the model to guess. Use those corrections to update your golden datasets and training data.

Real-World Example

A finance team is processing 500k+ invoices per month from hundreds of vendors. They started with a prompt-based extraction that worked in the pilot, then collapsed under real-world layouts: totals sometimes off by a few cents, line items missing on multi-page invoices, and one frequent failure on a specific carrier whose invoices are rotated scans with handwritten adjustments.

They move to a regression-tested pipeline:

1. Schema: They define a strict JSON Schema for invoices, including nested line items and enums for tax types.
2. Golden datasets: They curate 2,000 real invoices across their top 50 vendors, plus a special set of the problematic carrier’s documents. Operators use a review UI to mark correct totals, line items, and tax fields.
3. Versioned function: They create functions.extract_invoice_v1 on Bem, using Route -> Transform -> Enrich -> Validate primitives and set per-field confidence thresholds.
4. Automated evals: Every time they tweak the prompt, adjust routing, or switch to a new underlying model, Bem re-runs all 2,000 invoices and computes F1 per field and vendor.
5. Gates: They decide:
   • Global invoice F1 must stay above 0.99.
   • That problematic carrier’s invoices must stay above 0.995 on total_amount and line_items.quantity.
6. Upgrade attempt: A new model version improves extraction on handwritten notes but reduces recall on one vendor’s line items. The evals catch it: F1 on that vendor drops from 0.992 to 0.968, failing the gate. The team:
   • Keeps the existing function in production.
   • Uses regression results to craft a targeted fix.
   • Re-tests before a second rollout attempt.
7. Human review loop: Any low-confidence totals are routed to a Bem Surface. Corrections automatically fine-tune the function. New invoices from the problematic carrier gradually move from the review queue to fully automated processing as F1 improves and holds stable in regression tests.

Operationally, they move from a world where “every model change is a gamble” to one where they can show a CFO: “Last 24 hours we had 98.7% pass rate across all vendors, +1.2% vs last week. No regressions on priority carriers.”

Pro Tip: When you build your first golden dataset, over-index on the ugly edge cases—rotated scans, multi-doc packets, handwritten notes, credits, and exceptions. If a change passes those, it’s unlikely to blindside you in production.

Summary

Regression testing for document extraction is about turning stochastic behavior into predictable infrastructure. You define the contract with a schema, lock in golden datasets from real documents, version every function and workflow, and re-run historical payloads on every change. You measure precision, recall, and F1 at the field level, enforce hard gates, and route low-confidence cases into human review instead of pushing guesses into your ERP or product.

This isn’t “AI that magically stays accurate.” It’s evals, regression testing, and drift detection wired into your pipeline so a model or prompt change never quietly breaks production accuracy overnight.

Next Step

Get Started