🤖 AI Summary
This study addresses the degradation of model performance caused by noisy labels in crowdsourced annotation. We propose a robust learning framework grounded in signal processing principles, modeling annotator behavior and label generation mechanisms. For the first time, we introduce tensor identifiability and nonnegative matrix factorization (NMF) into crowdsourced truth inference, systematically tackling label fusion and latent ground-truth estimation. Methodologically, we unify statistical modeling, tensor decomposition, NMF, and preference-based learning techniques (e.g., RLHF and DPO), designing multiple noise-robust algorithms with provable convergence and mechanistic interpretability. Evaluated on diverse crowdsourced benchmark datasets, our approach achieves substantial improvements: +12.7% average accuracy in ground-truth recovery and +8.3% average gain in downstream classification accuracy. This work establishes a novel theoretical foundation and provides practical tools for learning from noisy labels.
📝 Abstract
One of the primary catalysts fueling advances in artificial intelligence (AI) and machine learning (ML) is the availability of massive, curated datasets. A commonly used technique to curate such massive datasets is crowdsourcing, where data are dispatched to multiple annotators. The annotator-produced labels are then fused to serve downstream learning and inference tasks. This annotation process often creates noisy labels due to various reasons, such as the limited expertise, or unreliability of annotators, among others. Therefore, a core objective in crowdsourcing is to develop methods that effectively mitigate the negative impact of such label noise on learning tasks. This feature article introduces advances in learning from noisy crowdsourced labels. The focus is on key crowdsourcing models and their methodological treatments, from classical statistical models to recent deep learning-based approaches, emphasizing analytical insights and algorithmic developments. In particular, this article reviews the connections between signal processing (SP) theory and methods, such as identifiability of tensor and nonnegative matrix factorization, and novel, principled solutions of longstanding challenges in crowdsourcing -- showing how SP perspectives drive the advancements of this field. Furthermore, this article touches upon emerging topics that are critical for developing cutting-edge AI/ML systems, such as crowdsourcing in reinforcement learning with human feedback (RLHF) and direct preference optimization (DPO) that are key techniques for fine-tuning large language models (LLMs).