Can You Trust the Confidence? ConfBench for Vision-Language Models on Document Extraction

2026-08-03

AI Summary

Existing document benchmarks lack low-quality samples, hindering reliable evaluation of visual language models’ confidence calibration for key information extraction and thereby impeding effective routing decisions between automated processing and human review. To address this gap, this work introduces ConfBench—the first confidence calibration benchmark tailored to this task—comprising 1,346 document variants generated via 20 controllable degradation procedures and over 70,000 entity annotations spanning the full spectrum of extraction accuracy. We propose the ECARB metric to translate discrimination performance into tangible human review cost savings and systematically evaluate multimodal inputs, confidence estimation strategies, and post-hoc calibration methods. Experiments reveal that combining OCR with image modality enhances confidence accuracy, log-probability aggregation of the first token yields optimal results, model capability—not parameter count—primarily governs calibration quality, and post-processing significantly improves threshold-based routing without compromising ranking performance.

Abstract

Intelligent document processing (IDP) with vision-language models (VLMs) hinges on confidence scores trustworthy enough to route extractions between automation and human review. Existing document benchmarks are dominated by clean, high-quality samples, leaving low accuracy regions too sparse for calibration assessment. We introduce ConfBench, the first calibration-specific benchmark for key information extraction (KIE), built by applying 20 controlled degradation pipelines to a diverse document set, yielding 1,346 variants and 70K+ entity-level evaluations spanning the full accuracy spectrum. We evaluate four proprietary and three open-weight VLMs under verbalized and log-probability confidence estimation methods across three input modalities, and find: (i) OCR+Image modality results in more accurate confidence estimates; (ii) model capability is the dominant factor: within the Claude family confidence quality scales monotonically with capability, while across families parameter count is a poor predictor; (iii) calibration quality varies widely across models, from near-perfect to severely overconfident, and per-model post-hoc correction rescales these absolute confidence values for threshold-based routing without altering ranking-based operational metrics; and (iv) log-probability with first-token aggregation consistently outperforms mean-token and margin aggregations. We also introduce ECARB, a review-budget metric translating discriminative gains into operational savings. We release ConfBench publicly to enable systematic study of confidence estimators and calibration methods for trustworthy IDP application deployment.

Problem

• confidence calibration
• vision-language models
• document extraction
• key information extraction
• intelligent document processing

Innovation

• ConfBench
• calibration
• vision-language models
• key information extraction
• confidence estimation