VELA: An LLM-Hybrid-as-a-Judge Approach for Evaluating Long Image Captions

2025-09-30

AI Summary

Existing automatic evaluation metrics are designed for short image captions and thus fail to accurately assess long, detailed descriptions generated by multimodal large language models (MLLMs). Moreover, mainstream LLM-as-a-Judge approaches suffer from low efficiency and high bias due to autoregressive inference and early fusion of visual information. To address these limitations, we propose VELA—a novel, efficient framework for evaluating long image descriptions—introducing the first LLM-Hybrid-as-a-Judge paradigm. VELA delays visual fusion, employs multi-stage reasoning, and aligns with multi-perspective human annotations across three dimensions: Descriptiveness, Relevance, and Fluency. We further introduce LongCap-Arena, the first benchmark dedicated to long caption evaluation, comprising nearly 8,000 images and tens of thousands of high-quality human annotations. Experiments demonstrate that VELA significantly outperforms existing metrics on LongCap-Arena, achieving exceptional agreement with human judgments (Spearman’s ρ > 0.92) and, for the first time, surpassing human-level performance in automatic long-description evaluation.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models
• Natural Language Processing: (Large) Language Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

In this study, we focus on the automatic evaluation of long and detailed image captions generated by multimodal Large Language Models (MLLMs). Most existing automatic evaluation metrics for image captioning are primarily designed for short captions and are not suitable for evaluating long captions. Moreover, recent LLM-as-a-Judge approaches suffer from slow inference due to their reliance on autoregressive inference and early fusion of visual information. To address these limitations, we propose VELA, an automatic evaluation metric for long captions developed within a novel LLM-Hybrid-as-a-Judge framework. Furthermore, we propose LongCap-Arena, a benchmark specifically designed for evaluating metrics for long captions. This benchmark comprises 7,805 images, the corresponding human-provided long reference captions and long candidate captions, and 32,246 human judgments from three distinct perspectives: Descriptiveness, Relevance, and Fluency. We demonstrated that VELA outperformed existing metrics and achieved superhuman performance on LongCap-Arena.

Problem

• Evaluating long detailed image captions from multimodal LLMs
• Overcoming limitations of short-caption metrics and slow LLM-judge inference
• Creating benchmark and metric for comprehensive long-caption assessment

Innovation

• Hybrid LLM framework for caption evaluation
• Late fusion technique to accelerate inference
• Specialized benchmark with multi-perspective human judgments