Measuring What Matters Beyond Text: Evaluating Multimodal Summaries by Quality, Alignment, and Diversity

2026-05-12

AI Summary

Existing evaluation methods struggle to jointly assess textual quality, image-text alignment, and visual diversity in multimodal summarization. To address this limitation, this work proposes MM-Eval, a unified evaluation framework that integrates these three dimensions into an interpretable, cohesive system, thereby overcoming the constraints of unimodal metrics. MM-Eval combines OpenFactScore, G-Eval, MLLM-as-a-judge, and Truncated CLIP Entropy, and leverages a regression model trained on the mLLM-EVAL benchmark to align with human preferences. Experimental results demonstrate that MM-Eval significantly outperforms heuristic aggregation approaches in predicting human judgments. Furthermore, the analysis reveals that factual consistency is the primary driver of overall quality, while image-text relevance and visual diversity provide complementary signals essential for comprehensive evaluation.

Abstract

Multimodal Large Language Models (MLLMs) have facilitated Multimodal Summarization with Multimodal Output (MSMO), wherein systems generate concise textual summaries accompanied by salient visuals from multimodal sources. However, current MSMO evaluation remains fragmented: text quality, image-text alignment, and visual diversity are typically assessed in isolation using unimodal metrics, making it difficult to capture whether the modalities jointly support a faithful and useful summary. To address this gap, we introduce MM-Eval, a unified evaluation framework that integrates assessments of textual quality, cross-modal alignment, and visual diversity. MM-Eval comprises three components: (1) text quality, measured using OpenFActScore for factual consistency and G-Eval for coherence, fluency, and relevance; (2) image-text relevance, evaluated via an MLLM-as-a-judge approach; and (3) image-set diversity, quantified using Truncated CLIP Entropy. We calibrate MM-Eval through a learned aggregation model trained on the mLLM-EVAL news benchmark, aligning component contributions with human preferences. Our analysis reveals a text-dominant hierarchy in this setting, where factual consistency acts as a critical determinant of perceived overall quality, while visual relevance and diversity provide complementary signals. MM-Eval improves over heuristic aggregation baselines and provides an interpretable, reference-weak framework for comparative evaluation of multimodal summaries.

Problem

• Multimodal Summarization
• Evaluation Framework
• Cross-modal Alignment
• Visual Diversity
• Text Quality

Innovation

• multimodal summarization
• unified evaluation framework
• cross-modal alignment
• visual diversity
• MLLM-as-a-judge