ORAV: Benchmarking Audio-Video Generation from Multimodal Contexts

2026-09-28

AI Summary

This study addresses the challenges of compositional control and contextual understanding in audio-visual generation under heterogeneous multimodal references by proposing a systematic evaluation framework. Methodologically, we construct ORAV, a benchmark comprising 380 instances, and design a reference-aware pairwise evaluation protocol that integrates multimodal instruction parsing, audio-visual evidence preparation, and bidirectional ordinal consistency verification. Our investigation reveals combinatorial discrepancies and unexpected content replication deficiencies that are obscured by conventional holistic ranking approaches. Achieving an 86.08% agreement with human judgment, this protocol provides reproducible diagnostic dimensions for assessing both quality and affinity in multi-reference audio-visual generation.

Abstract

Audio-video generation using heterogeneous multimodal references has emerged as a new challenge, requiring both compositional control over generation and grounded understanding of multimodal context. In this paper, we introduce ORAV Bench for Omni Reference Audio-Video Generation, comprising 380 task instances with 2-10 references, 9 semantic roles, and 30 role compositions. Instructions specify the relationships among references; the media supply the identities, dynamics, and audio characteristics to be realized. To evaluate these open-ended outputs, we develop a reference-aware pairwise protocol that prepares visual and auditory evidence, compares the intended contribution of each reference, and checks the overall verdict in both presentation orders. On held-out instances, it achieves 86.08% effective agreement with human judgments. Across 5 frontier systems, overall rankings conceal distinct strengths across reference compositions. A recurring failure is to reproduce unintended source content in place of the requested result, despite closely resembling a reference. Reproducible pointwise diagnostics of quality, reference affinity, and speech reveal distinct dimensions of model behavior. ORAV thus offers a benchmark for tracking progress toward controllable, compositional, and reference-faithful audio-video generation.

Problem

• audio-video generation
• multimodal context
• compositional control
• benchmark
• reference-faithful

Innovation

• Audio-Video Generation
• Multimodal Benchmark
• Reference-Aware Evaluation
• Compositional Control
• Pairwise Protocol