A Model for Imbalanced Label Aggregation: A Focus on Minority-Class Detection

2026-07-27

AI Summary

This work addresses the challenge of accurately recovering minority-class labels in crowdsourced annotation tasks under severe class imbalance. The authors propose a generative label aggregation model that jointly captures category-dependent item difficulty and annotator competence, departing from conventional assumptions in the field. By re-examining the Condorcet jury theorem under class-imbalanced settings, they theoretically demonstrate that majority voting asymptotically preserves the original class distribution. Empirical evaluation across 33 real-world multiclass crowdsourcing datasets shows that the proposed model substantially improves recall for minority classes while maintaining competitive overall accuracy.

Abstract

We study imbalanced crowdsourcing with a focus on class-dependent annotator accuracy, a setting that, to the best of our knowledge, remains relatively underexplored despite its importance in real-world inspection systems where the labels of greatest operational importance are also the rarest ones. In this setting, annotators may be reliable on both classes, unreliable on both classes, majority-class specialists, or minority-class specialists. Existing models only partially address this problem: they either capture class-dependent errors but ignore item difficulty, or they model item difficulty without capturing class-dependent errors. To fill this gap for imbalanced datasets in crowdsourcing, we introduce a generative aggregation model combining item difficulty with class-dependent annotator competence. The model allows both annotator abilities and item difficulties to vary across classes. We then revisit Condorcet's Jury Theorem in the class-imbalanced setting. We also show that majority voting asymptotically preserves the underlying class proportion. We evaluate our model on $33$ real-world crowdsourcing datasets, covering multiclass tasks such as images and text, as well as two large-scale regimes: large-scale annotation datasets, with many annotations per item, and large-scale item datasets, with a large number of annotated instances. Across these diverse settings, our model consistently achieves the highest minority recall while remaining competitive in balanced accuracy, making it particularly relevant when rare-label recovery is the primary objective.

Problem

• imbalanced crowdsourcing
• class-dependent annotator accuracy
• minority-class detection
• label aggregation
• item difficulty

Innovation

• imbalanced crowdsourcing
• class-dependent annotator accuracy
• item difficulty
• minority-class detection
• generative aggregation model