Multiplicative Logit Adjustment Approximates Neural-Collapse-Aware Decision Boundary Adjustment

2024-09-26

arXiv.org

AI Summary

To address classifier bias toward tail classes in long-tailed recognition, this paper establishes, for the first time, a rigorous equivalence between multiplicative logit adjustment (MLA) and optimal decision boundary calibration—grounded in the neural collapse phenomenon. We prove that MLA approximates the theoretically optimal boundary adjustment induced by neural collapse. Methodologically, we model collapsed feature distributions and derive a closed-form solution for the optimal boundary, revealing MLA as a parameter-efficient approximation thereof. Extensive experiments on major long-tailed benchmarks—including ImageNet-LT and iNaturalist—demonstrate MLA’s effectiveness and robustness, yielding substantial gains in tail-class accuracy. We further provide reproducible, practical guidelines for hyperparameter tuning. Our core contribution lies in theoretically elucidating the fundamental reason behind MLA’s empirical success, thereby bridging neural collapse theory with practical logit adjustment in long-tailed learning.

Technology Category

• Machine Learning: Multi-class/Multi-label Learning & Extreme Classification
• Natural Language Processing: Learning & Optimization for NLP
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: ML for personalized search and recommendations
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling

Abstract

Real-world data distributions are often highly skewed. This has spurred a growing body of research on long-tailed recognition, aimed at addressing the imbalance in training classification models. Among the methods studied, multiplicative logit adjustment (MLA) stands out as a simple and effective method. What theoretical foundation explains the effectiveness of this heuristic method? We provide a justification for the effectiveness of MLA with the following two-step process. First, we develop a theory that adjusts optimal decision boundaries by estimating feature spread on the basis of neural collapse. Second, we demonstrate that MLA approximates this optimal method. Additionally, through experiments on long-tailed datasets, we illustrate the practical usefulness of MLA under more realistic conditions. We also offer experimental insights to guide the tuning of MLA hyperparameters.

Problem

• Addresses imbalance in training classification models
• Explains effectiveness of multiplicative logit adjustment
• Adjusts decision boundaries using neural collapse theory

Innovation

• Multiplicative Logit Adjustment for decision boundaries
• Neural collapse theory guides feature spread estimation
• Hyperparameter tuning insights for long-tailed datasets