Confidence Estimation for Financial Vision-Language Models in Chart and Document Understanding

2026-08-06

AI Summary

In financial visual question answering (VQA), large vision-language models (LVLMs) often produce fluent yet hallucinated responses unsupported by the underlying charts, posing significant decision-making risks. This work systematically evaluates seven confidence estimation methods—three inference-time baselines and four trained probes—across five open-source LVLMs and four financial VQA tasks, with all probes trained exclusively on natural images to assess cross-domain generalization. The study reveals that well-calibrated confidence is rarer than reliable ranking performance and that reliability varies substantially across model–task pairs. It introduces an actionable, error-budget-based automation threshold and proposes, for the first time, a grounding-aware probe that effectively distinguishes fluent but unfounded guesses from chart-grounded answers. Results show that only trained probes yield reliable calibration scores, enabling partial automation under lenient conditions, yet high-difficulty tasks remain largely non-automatable within a 5% error budget.

Abstract

LVLMs are increasingly used to read financial charts, tables, and documents, where a single misread figure can move a decision and the most authoritative-looking answer is sometimes one the model produced without reading the exhibit. The operational question is therefore trust, not accuracy: which answers can be acted on, and which escalated to a reviewer. We evaluate seven confidence estimators, three inference-only and four trained internal probes, across five open-weight LVLMs and four conditions from three financial visual question-answering benchmarks, one bilingual; every probe is trained only on natural images and applied to finance without adaptation, so the results measure out-of-distribution transfer. Three findings hold. First, the scarce property is calibration, not ranking: the inference baselines rank correct above incorrect answers competitively but are badly overconfident, calibration error far above what a threshold can tolerate, and only the trained probes produce a thresholdable score. Second, reliability is structured rather than global, along two axes a practitioner can read directly: the best estimator shifts with both model and task, none leading more than eight of twenty (model, condition) cells, and a controlled bilingual contrast exposes an apparent language robustness as a composition artifact that dissolves once models are read one at a time. Third, cast as deferral under an error budget, how much can be safely automated is set first by the model's competence and only narrowed by its confidence, so deferral clears a real share of the easiest condition and almost none of the hardest, near zero at a strict 5% budget. Two trained probes carry the calibration a deferral policy needs, and among them only the grounding-aware one lowers its confidence on answers a model gives without using the figure, separating detected non-grounding from a fluent guess.

Problem

• confidence estimation
• financial vision-language models
• trust
• calibration
• deferral

Innovation

• confidence estimation
• vision-language models
• calibration
• deferral policy
• grounding-aware