Neural Collapse-Inspired Multi-Label Federated Learning under Label-Distribution Skew

2025-09-15

AI Summary

In federated learning, multi-label data distribution skew—arising from label co-occurrence, inter-label dependencies, and inconsistencies between local and global label structures—severely degrades model performance. Method: This paper presents the first systematic study of multi-label federated learning under label skew, proposing a neural collapse (NC)-inspired feature disentanglement framework. It introduces shared NC-guided latent-space clustering and jointly employs a feature disentanglement module with NC regularization loss to achieve tight, semantics-specific feature alignment across heterogeneous clients, thereby mitigating model conflicts caused by inconsistent inter-class relationships. Results: Extensive experiments on four benchmark datasets under eight distinct skew settings demonstrate significant improvements over state-of-the-art methods, validating the framework’s effectiveness and robustness to diverse multi-label distribution skews.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Natural Language Processing: Safety and Robustness

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Federated Learning (FL) enables collaborative model training across distributed clients while preserving data privacy. However, the performance of deep learning often deteriorates in FL due to decentralized and heterogeneous data. This challenge is further amplified in multi-label scenarios, where data exhibit complex characteristics such as label co-occurrence, inter-label dependency, and discrepancies between local and global label relationships. While most existing FL research primarily focuses on single-label classification, many real-world applications, particularly in domains such as medical imaging, often involve multi-label settings. In this paper, we address this important yet underexplored scenario in FL, where clients hold multi-label data with skewed label distributions. Neural Collapse (NC) describes a geometric structure in the latent feature space where features of each class collapse to their class mean with vanishing intra-class variance, and the class means form a maximally separated configuration. Motivated by this theory, we propose a method to align feature distributions across clients and to learn high-quality, well-clustered representations. To make the NC-structure applicable to multi-label settings, where image-level features may contain multiple semantic concepts, we introduce a feature disentanglement module that extracts semantically specific features. The clustering of these disentangled class-wise features is guided by a predefined shared NC structure, which mitigates potential conflicts between client models due to diverse local data distributions. In addition, we design regularisation losses to encourage compact clustering in the latent feature space. Experiments conducted on four benchmark datasets across eight diverse settings demonstrate that our approach outperforms existing methods, validating its effectiveness in this challenging FL scenario.

Problem

• Addresses multi-label federated learning with skewed label distributions
• Proposes feature alignment and disentanglement for improved representation learning
• Mitigates performance degradation from decentralized heterogeneous multi-label data

Innovation

• Feature disentanglement module for multi-label semantics
• Predefined shared Neural Collapse structure alignment
• Regularization losses for compact latent clustering