Mixup Helps Understanding Multimodal Video Better

2025-10-12

AI Summary

In multimodal video understanding, models often overfit to dominant modalities while suppressing contributions from weaker ones, degrading generalization. To address this, we propose a feature-level dynamic mixing framework: first, multimodal Mixup (MM) is applied *after* multimodal feature fusion to generate synthetic feature-label pairs; second, a balancing strategy (B-MM) dynamically adjusts mixing weights based on each modality’s real-time contribution, mitigating modality imbalance. This work is the first to apply Mixup in the post-fusion feature space and enables adaptive, contribution-aware modality weighting. Extensive experiments across multiple video understanding benchmarks demonstrate significant improvements in model generalization and robustness—particularly under modality corruption or distribution shift—validating the efficacy of our approach for complex multimodal tasks.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems

Abstract

Multimodal video understanding plays a crucial role in tasks such as action recognition and emotion classification by combining information from different modalities. However, multimodal models are prone to overfitting strong modalities, which can dominate learning and suppress the contributions of weaker ones. To address this challenge, we first propose Multimodal Mixup (MM), which applies the Mixup strategy at the aggregated multimodal feature level to mitigate overfitting by generating virtual feature-label pairs. While MM effectively improves generalization, it treats all modalities uniformly and does not account for modality imbalance during training. Building on MM, we further introduce Balanced Multimodal Mixup (B-MM), which dynamically adjusts the mixing ratios for each modality based on their relative contributions to the learning objective. Extensive experiments on several datasets demonstrate the effectiveness of our methods in improving generalization and multimodal robustness.

Problem

• Addresses overfitting to dominant modalities in video understanding
• Proposes balanced mixing to handle modality imbalance dynamically
• Improves generalization and robustness in multimodal learning

Innovation

• Mixup applied to multimodal feature level
• Balanced Mixup adjusts modality mixing ratios
• Dynamic ratio adjustment based on modality contributions