CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

2026-08-04

AI Summary

This work addresses the limited scope of existing video-language model evaluations, which predominantly focus on short clips and single-sentence captions, thereby lacking effective assessment of paragraph-level comprehension of long-form videos. To bridge this gap, we introduce CLIP-CC-Bench, a new benchmark comprising five hours of movie content, 90-second segments, and expert-authored paragraph-level descriptions. We propose the first evaluation framework tailored for paragraph-level video captioning, integrating multiple LLM-based embedding models, coarse- and fine-grained semantic matching, Borda count aggregation, and bootstrap resampling to significantly enhance evaluation stability and reliability. Using this framework, we conduct a systematic assessment of 17 prominent video-language models, yielding a highly consistent overall ranking. All tools and resources are publicly released to support reproducible research.

Abstract

Benchmarking video-language models has largely focused on short clips and single-sentence metrics, leaving open whether current systems can generate accurate long-form, paragraph-level descriptions. We introduce CLIP-CC-Bench, an evaluation suite for long-form video description built from 5 hours of movie content segmented into 90-second clips, each paired with an expert-written paragraph-style reference. The evaluation suite employs an ensemble of five state-of-the-art LLM-based embedding models to increase reliability and mitigate single-model bias, and applies two complementary methodologies: (i) coarse-grained semantic matching and (ii) fine-grained semantic matching to compare model-generated descriptions against CLIP-CC-Bench references. Using this framework, we evaluate 17 state-of-the-art video-language models and report both their Borda-aggregated rankings and their average scores on CLIP-CC-Bench. We further quantify the protocol's internal reliability through inter-judge agreement and bootstrap ranking stability. We release standardized evaluation scripts, model outputs, and aggregation tools at https://github.com/Multimodal-Intelligence-Lab/CLIP-CC-Bench to support reproducibility. CLIP-CC-Bench provides a practical evaluation framework for long-form video description, filling a gap left by existing short-clip and QA-only benchmarks.

Problem

• video-language models
• paragraph-level description
• long-form video captioning
• benchmarking
• evaluation

Innovation

• paragraph-level video description
• video-language benchmarking
• LLM-based embedding ensemble
• semantic matching
• long-form video understanding