AVCap: Reinforcing Audio-Video Joint Caption with Detail-Aware Reward

2026-08-07

AI Summary

This work addresses three key challenges in fine-grained audio-visual joint captioning: the scarcity of high-quality data, coarse rewards in reinforcement learning, and the absence of atomic-level evaluation benchmarks. To tackle these issues, the authors propose the AVCap framework, which introduces AVCap-100K—a dataset of 100,000 temporally aligned, detail-rich audio-visual descriptions—alongside a detail-aware GRPO reinforcement learning algorithm and a multimodal fusion architecture. Additionally, they release AVCap-Bench, the first benchmark for atomic-level caption evaluation, accompanied by the AVCap-Score metric. Experimental results demonstrate that the open-source AVCap model achieves state-of-the-art performance in both detail accuracy and semantic richness, matching or even surpassing several closed-source models across multiple metrics.

Abstract

Detailed audio-video joint captioning is essential for multimodal video understanding and generation. However, prior works are constrained by three main limitations: (1) the scarcity of high-quality public datasets with fine-grained audio-visual joint captions; (2) reinforcement-learning methods that rely on coarse reward signals; and (3) the lack of a benchmark and metric for evaluating detailed audiovisual captions at the atomic level. To address these challenges, we propose: (1) AVCap-100K, a high-quality dataset of 100K temporally aligned, detail-rich audio-video captions; (2) AVCap, a model optimized via Detail-Aware GRPO (Da-GRPO) that achieves state-of-the-art performance among open-source models and matches or surpasses proprietary models on several evaluations; and (3) AVCap-Bench and AVCap-Score, a specialized benchmark and metric for evaluating atomic-level details in audiovisual captions. Our code, models, and datasets are available at https://huggingface.co/collections/Apryle/avcap.

Problem

• audio-video joint captioning
• fine-grained dataset
• reinforcement learning
• evaluation benchmark
• detail-aware reward

Innovation

• audio-video joint captioning
• detail-aware reinforcement learning
• AVCap-100K dataset
• atomic-level evaluation
• Da-GRPO