SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

2026-07-29

AI Summary

This study addresses the limitations of existing image quality assessment methods in effectively evaluating scientific figure quality, particularly their inability to jointly assess caption consistency, citation relevance, and visual misleadingness. To bridge this gap, the work introduces a fine-grained evaluation benchmark that models figures in conjunction with their captions, citing sentences, and full-paper context across five dimensions: clarity, layout, caption alignment, contextual relevance, and risk of misinterpretation. A high-quality expert-annotated dataset is curated to support this benchmark. Building upon it, the authors propose SFQ-Agent, an auditable multimodal evaluation framework that integrates large language models and vision-language models for cross-modal reasoning. On the eval1200 test set, SFQ-Agent (F3), grounded in GPT-5.6-Sol, achieves a mean absolute error of 0.418 and a consistency rate of 93.4%, significantly outperforming current approaches.

Abstract

Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers top computer-science conferences from 2020 to 2025; 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation framework SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.

Problem

• scientific figure quality
• image quality assessment
• full-manuscript context
• caption alignment
• visual misleadingness

Innovation

• scientific figure quality assessment
• full-manuscript context
• cross-modal evaluation
• misleading risk detection
• benchmark dataset