All modalities are equal, but video is more equal: Closing the Cross-Attention Gap in Joint Video Generation

2026-09-23

AI Summary

研究解决了多模态生成中视频与其他模态之间的不对称对应问题，通过引入RecCAR方法来增强非视频模态对视频的约束力，从而提升生成质量。

Abstract

Video is a rich representation of a physical event, capturing appearance, geometry, motion, and temporal evolution. Other modalities, such as 3D body motion or audio, encode narrower aspects of the same event. We find that joint multimodal diffusion transformers exhibit a corresponding asymmetry in cross-modal correspondence: companion modalities develop strong correspondences to video, but the reciprocal correspondences through which they constrain video remain substantially weaker. We express both directions as comparable correspondence distributions over video tokens and define their disagreement as the reciprocal correspondence gap. We introduce RecCAR, standing for Reciprocal Cross-modal Attention Regularization, a KL regularizer that uses the well-established video-to-modality correspondence as a fixed reference and aligns the weaker modality-to-video correspondence toward it. Across joint video-motion and video-audio generation, RecCAR improves the Human Anatomy score from 0.69 to 0.75 and reduces audio-video desynchronization from 0.804 to 0.752, while improving overall generation

Problem

• Cross-Attention Gap
• Multimodal Diffusion Transformers
• Reciprocal Correspondence

Innovation

• Reciprocal Cross-modal Attention
• KL Regularizer
• Multimodal Diffusion Transformers