Rigel: Self-Distilled Score Adaptation for Image and Video Captioning Evaluation

2026-06-29

AI Summary

Existing automatic evaluation metrics exhibit insufficient alignment with human judgments of image and video caption quality, particularly in tasks with small label sets where large language model (LLM)-based approaches underperform. To address this limitation, this work proposes Rigel, an evaluation method based on self-distillation with score adaptation. Rigel distills a lightweight scoring head tailored for evaluation from a frozen LLM and fine-tunes the backbone model using human judgment data, thereby transferring LLM capabilities into a task-aligned, low-dimensional scoring space without relying on large-vocabulary outputs. Evaluated across multiple benchmarks, Rigel substantially outperforms existing metrics, achieving over a 10-point improvement in correlation under the no-reference setting on ActivityNet-Fact and significantly enhancing consistency with human assessments.

Abstract

Automatic evaluation of image and video captioning is essential for benchmarking multimodal systems, although standard evaluation metrics show limited alignment with human judgments. Recent approaches using large language models (LLMs), commonly referred to as LLM-as-a-Judge, have improved alignment with human judgments but still suffer from a mismatch between large-vocabulary language modeling and evaluation over a small label set. To address this, we propose Rigel, an automatic evaluation metric for image and video captioning, based on self-distilled score adaptation. The metric employs an evaluation-specific scoring head distilled from a frozen LLM, which captures judgment signals in a task-aligned space without relying on large-vocabulary token sets. We then refine the LLM backbone with human judgment data. To train Rigel, we constructed the Vid-Lepus dataset, which contains 3,338 video clips, 33,380 reference captions, and 5,637 candidate captions. Experiments on multiple benchmarks show that Rigel outperforms state-of-the-art metrics, achieving over 10-point improvements on ActivityNet-Fact in the reference-free setting.

Problem

• image captioning evaluation
• video captioning evaluation
• LLM-as-a-Judge
• human judgment alignment
• automatic evaluation metrics

Innovation

• self-distilled score adaptation
• LLM-as-a-Judge
• caption evaluation
• task-aligned scoring head
• Vid-Lepus dataset