SciFigAlign: Scoring Scientific Figures by Fine-tuned Alignment of Visuals with Manuscript Evidence

2026-07-29

AI Summary

Existing image quality assessment methods struggle to evaluate whether scientific figures effectively support their associated paper claims, primarily due to insufficient modeling of visual-textual evidence consistency and scientific context. To address this, this work introduces SciFigAlign, a novel model that leverages CLIP and SciBERT with cross-modal attention and a CubeMLP fusion mechanism, jointly optimizing regression and ranking losses using figures, captions, citation contexts, and lightweight surrounding text. The study constructs a dataset of 3,857 peer-reviewed figures and proposes the first evidence-anchored figure quality assessment framework, featuring a peer-review-driven four-dimensional scoring system. Experiments on a paper-level test set (n=396) demonstrate that SciFigAlign achieves a macro MAE of 0.3524 and pairwise accuracy of 81.64%, reducing relative error by 59% compared to the strongest LLM-as-judge baseline, thereby validating the superiority of evidence-aligned training over prompt-only paradigms.

Abstract

Scientific figure assessment in peer review differs fundamentally from general image quality evaluation: a figure must be visually legible, faithfully support the manuscript's claims, and communicate evidence with a clear visual hierarchy. However, if we apply traditional image assessment methods to scientific figure quality assessment, limitations emerge: classic IQA models capture perceptual quality or aesthetics but cannot judge whether a figure serves the paper's scientific argument; CLIP-based methods assess generic image-text correspondence, yet lack understanding of manuscript context; and zero-shot LLM/VLM judges, when repurposed for figure scoring, often yield overly concentrated scores with limited fusion of visual and textual evidence. We introduce an annotated dataset of 3,857 scientific figures from peer-reviewed conference papers, each rated along four peer-review-oriented dimensions: Clarity, Relevance, Informativeness, and Structure. We propose SciFigAlign, a fine-tuned multimodal scorer that grounds figure quality assessment in manuscript evidence. Given a figure crop, caption, citing paragraphs, and light paper context, SciFigAlign fine-tunes CLIP and SciBERT end-to-end with per-modality cross-attention and CubeMLP fusion, jointly optimizing SmoothL1 regression with a within-paper ranking hinge loss. Under paper-level splits, SciFigAlign achieves a macro MAE of 0.3524 and a within-paper pairwise accuracy of 81.64% on the test set with n = 396, a 59% relative error reduction over the best LLM-as-judge baseline with MAE 0.864. Ablations confirm that manuscript-grounded inputs, citing-context denoising, and ranking supervision are all critical, showing that scientific figure assessment requires learned alignment between visual content and manuscript evidence rather than prompting alone, even with state-of-the-art VLMs.

Problem

• scientific figure assessment
• manuscript evidence alignment
• peer review
• multimodal scoring
• visual-textual grounding

Innovation

• scientific figure assessment
• multimodal alignment
• manuscript-grounded evaluation
• CLIP-SciBERT fusion
• peer-review scoring