🤖 AI Summary
This study investigates how to allocate annotation resources effectively according to target evaluation metrics to capture annotator disagreement. Leveraging the ChaosNLI dataset and fine-tuning DeBERTa and RoBERTa models with soft labels, the work reveals for the first time that the annotation saturation point is highly metric-dependent: distribution matching saturates at around 10 annotators (achieving 87–95% of maximal performance gain), whereas disagreement detection requires 20–50 annotators. The experiments demonstrate that soft labels effectively distinguish ambiguous from unambiguous examples, achieving an entropy correlation of 0.643—significantly outperforming conventional label smoothing methods (0.45–0.49). These findings underscore the necessity and efficacy of adopting evaluation-metric-driven annotation strategies.
📝 Abstract
When annotators disagree on a label, the disagreement itself carries signal -- and the number of annotators needed to capture it depends on the evaluation metric. We fine-tune NLI models on label distributions subsampled from ChaosNLI, a dataset providing 100 independent annotator judgments per item, and identify metric-dependent saturation. In our 3-class NLI setting, entropy correlation -- whether the model identifies which items elicit disagreement -- requires N ~ 20-50 annotators to converge, while distributional match (KL divergence) saturates by N ~ 10 (87-95% of improvement across five model seeds). This finding rests on a prior observation: soft labels carry item-specific signal that label smoothing cannot replicate. Across five smoothing intensities, entropy correlation clusters at r ~ 0.45-0.49, while soft labels reach r = 0.643 (p < 0.001); per-item analysis traces this gap to smoothing's inability to distinguish ambiguous items from clear ones. The soft-label advantage replicates across two architectures (DeBERTa, RoBERTa), a non-NLI-pretrained baseline, and an exploratory cross-domain evaluation on content safety. These results suggest that annotation budgets should be informed by the target evaluation metric rather than set uniformly.