Elastic Representation: Mitigating Spurious Correlations for Group Robustness

2025-02-14

AI Summary

Deep learning models often suffer from spurious correlations between inputs and labels, severely degrading group robustness—particularly under data scarcity or class imbalance—leading to substantial performance drops on minority groups. To address this, we propose Elastic Representation (ElRep), a novel representation learning framework that jointly imposes nuclear norm and Frobenius norm constraints on the penultimate-layer representations of neural networks. This is the first application of elastic regularization to representation learning, explicitly balancing feature importance and diversity. Theoretically, we prove that ElRep minimizes in-distribution overall predictive degradation, overcoming a key limitation of existing fairness-aware methods that typically sacrifice aggregate accuracy for improved subgroup performance. Extensive experiments on multiple imbalanced benchmark datasets demonstrate that ElRep significantly improves minority-group accuracy while preserving or even enhancing overall accuracy, achieving state-of-the-art robust generalization.

Technology Category

• Machine Learning: Representation Learning
• Computer Vision: Representation Learning for Vision
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

Deep learning models can suffer from severe performance degradation when relying on spurious correlations between input features and labels, making the models perform well on training data but have poor prediction accuracy for minority groups. This problem arises especially when training data are limited or imbalanced. While most prior work focuses on learning invariant features (with consistent correlations to y), it overlooks the potential harm of spurious correlations between features. We hereby propose Elastic Representation (ElRep) to learn features by imposing Nuclear- and Frobenius-norm penalties on the representation from the last layer of a neural network. Similar to the elastic net, ElRep enjoys the benefits of learning important features without losing feature diversity. The proposed method is simple yet effective. It can be integrated into many deep learning approaches to mitigate spurious correlations and improve group robustness. Moreover, we theoretically show that ElRep has minimum negative impacts on in-distribution predictions. This is a remarkable advantage over approaches that prioritize minority groups at the cost of overall performance.

Problem

• Mitigating spurious correlations in deep learning
• Improving group robustness with Elastic Representation
• Balancing feature diversity and importance effectively

Innovation

• Nuclear-norm penalties for feature learning
• Frobenius-norm penalties on neural representations
• Elastic Representation mitigates spurious correlations