BalanceBenchmark: A Survey for Imbalanced Learning

2025-02-15

AI Summary

This paper addresses the problem of *modality imbalance*—uneven utilization of information across modalities—in multimodal learning. Methodologically, it introduces BalanceBenchmark, the first standardized evaluation benchmark specifically designed for this issue. It systematically categorizes existing algorithms into four distinct strategy families; establishes a three-dimensional evaluation framework encompassing accuracy, modality balance, and computational complexity; and develops a modular, extensible open-source toolkit integrated within the PyTorch ecosystem. Comprehensive experiments are conducted across multiple mainstream multimodal datasets under unified protocols, revealing fundamental trade-offs among performance, modality balance, and efficiency across algorithms. The codebase is publicly released and has been widely adopted by the research community. This work establishes a foundation for fair, reproducible, and principled investigation of modality imbalance in multimodal learning.

Technology Category

• Machine Learning: Multimodal Learning
• Computer Vision: Multi-modal Vision
• Natural Language Processing: Language Grounding & Multi-modal NLP

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Multimodal learning has gained attention for its capacity to integrate information from different modalities. However, it is often hindered by the multimodal imbalance problem, where certain modality dominates while others remain underutilized. Although recent studies have proposed various methods to alleviate this problem, they lack comprehensive and fair comparisons. In this paper, we systematically categorize various mainstream multimodal imbalance algorithms into four groups based on the strategies they employ to mitigate imbalance. To facilitate a comprehensive evaluation of these methods, we introduce BalanceBenchmark, a benchmark including multiple widely used multidimensional datasets and evaluation metrics from three perspectives: performance, imbalance degree, and complexity. To ensure fair comparisons, we have developed a modular and extensible toolkit that standardizes the experimental workflow across different methods. Based on the experiments using BalanceBenchmark, we have identified several key insights into the characteristics and advantages of different method groups in terms of performance, balance degree and computational complexity. We expect such analysis could inspire more efficient approaches to address the imbalance problem in the future, as well as foundation models. The code of the toolkit is available at https://github.com/GeWu-Lab/BalanceBenchmark.

Problem

• Addresses multimodal imbalance in learning
• Introduces BalanceBenchmark for fair evaluations
• Systematically categorizes imbalance mitigation strategies

Innovation

• Multimodal imbalance algorithms categorized
• BalanceBenchmark introduced for evaluation
• Modular toolkit standardized experimental workflow