Preserving Task-Relevant Information Under Linear Concept Removal

2025-06-12

AI Summary

Neural network representations often entangle sensitive attributes with task-relevant information, compromising fairness and interpretability. To address this, we propose a linear concept removal method that completely eliminates linearly predictable sensitive attributes while strictly preserving the covariance between representations and primary task labels—thereby ensuring lossless retention of task-relevant second-order statistical signal. Our key contribution is the construction of a unique optimal oblique projection operator, rigorously justified through covariance-preserving optimization and theoretical analysis of concept predictability elimination. Experiments on benchmarks including Bias in Bios and Winobias demonstrate over 95% removal rate of sensitive attributes, with less than 0.8% degradation in main-task accuracy—significantly outperforming existing post-processing debiasing methods.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Machine Learning: Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Modern neural networks often encode unwanted concepts alongside task-relevant information, leading to fairness and interpretability concerns. Existing post-hoc approaches can remove undesired concepts but often degrade useful signals. We introduce SPLICE-Simultaneous Projection for LInear concept removal and Covariance prEservation-which eliminates sensitive concepts from representations while exactly preserving their covariance with a target label. SPLICE achieves this via an oblique projection that"splices out"the unwanted direction yet protects important label correlations. Theoretically, it is the unique solution that removes linear concept predictability and maintains target covariance with minimal embedding distortion. Empirically, SPLICE outperforms baselines on benchmarks such as Bias in Bios and Winobias, removing protected attributes while minimally damaging main-task information.

Problem

• Removes unwanted concepts from neural networks
• Preserves task-relevant covariance with target labels
• Minimizes damage to main-task information during removal

Innovation

• Oblique projection removes unwanted concepts
• Preserves covariance with target label
• Minimizes embedding distortion theoretically