🤖 AI Summary
Existing benchmarks struggle to systematically evaluate the impact of multimodal inputs on structured information extraction. This work proposes the first multimodal extraction benchmark tailored for government forms, generating diverse documents with deterministic ground truth through procedural PDF templates and a reverse annotation pipeline. Each document is provided in four standardized input formats—plain text, layout-preserving text, image, and multimodal—to enable rigorous ablation studies. A three-stage quality control mechanism and compliance-based evaluation of structured outputs ensure high data fidelity and reliable assessment. Experiments reveal that small models (<4B parameters) are primarily limited by structural adherence; fine-tuning a 2B model yields an 81-percentage-point improvement; layout-preserving text consistently outperforms image inputs by 3–18 percentage points; and the benchmark exhibits maximal discriminative power within the 60–95% accuracy range.
📝 Abstract
We introduce VAREX (VARied-schema EXtraction), a benchmark for evaluating multimodal foundation models on structured data extraction from government forms. VAREX employs a Reverse Annotation pipeline that programmatically fills PDF templates with synthetic values, producing deterministic ground truth validated through three-phase quality assurance. The benchmark comprises 1,777 documents with 1,771 unique schemas across three structural categories, each provided in four input modalities: plain text, layout-preserving text (whitespace-aligned to approximate column positions), document image, or both text and image combined. Unlike existing benchmarks that evaluate from a single input representation, VAREX provides four controlled modalities per document, enabling systematic ablation of how input format affects extraction accuracy -- a capability absent from prior benchmarks. We evaluate 20 models from frontier proprietary models to small open models, with particular attention to models <=4B parameters suitable for cost-sensitive and latency-constrained deployment. Results reveal that (1) below 4B parameters, structured output compliance -- not extraction capability -- is a dominant bottleneck; in particular, schema echo (models producing schema-conforming structure instead of extracted values) depresses scores by 45-65 pp (percentage points) in affected models; (2) extraction-specific fine-tuning at 2B yields +81 pp gains, demonstrating that the instruction-following deficit is addressable without scale; (3) layout-preserving text provides the largest accuracy gain (+3-18 pp), exceeding pixel-level visual cues; and (4) the benchmark most effectively discriminates models in the 60-95% accuracy band. Dataset and evaluation code are publicly available.