Mitigating the Impact of Labeling Errors on Training via Rockafellian Relaxation

2024-05-30

arXiv.org

AI Summary

Label noise—arising from human annotation errors or weak supervision—severely degrades neural network performance, especially at high noise rates. To address this, we propose Rockafellian Relaxation Method (RRM), an architecture-agnostic robust training framework that introduces Rockafellian relaxation theory—a convex-analytic concept—into robust learning for the first time. RRM constructs an adaptive loss reweighting mechanism without modifying network architecture or assuming specific noise types, thereby unifying treatment of both stochastic and adversarial label corruption. Integrated into end-to-end training, RRM consistently improves accuracy across diverse tasks, including image classification and NLP sentiment analysis. It demonstrates strong robustness under high label noise (up to 60%) and adversarial perturbations, outperforming state-of-the-art robust training methods.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Labeling errors in datasets are common, arising in a variety of contexts, such as human labeling, noisy labeling, and weak labeling (i.e., image classification). Although neural networks (NNs) can tolerate modest amounts of these errors, their performance degrades substantially once error levels exceed a certain threshold. We propose a new loss reweighting, architecture-independent methodology, Rockafellian Relaxation Method (RRM) for neural network training. Experiments indicate RRM can enhance neural network methods to achieve robust performance across classification tasks in computer vision and natural language processing (sentiment analysis). We find that RRM can mitigate the effects of dataset contamination stemming from both (heavy) labeling error and/or adversarial perturbation, demonstrating effectiveness across a variety of data domains and machine learning tasks.

Problem

• Reduces labeling errors impact
• Enhances neural network robustness
• Mitigates dataset contamination effects

Innovation

• Rockafellian Relaxation Method
• Loss reweighting technique
• Robust performance enhancement