SHALA-LLM: Smartly Handling Ambiguous Labels in Aligning LLMs

2026-06-03

AI Summary

This work addresses a critical limitation in existing large language model alignment methods, which typically disregard the disagreement among human annotators on ambiguous samples and thus fail to capture label ambiguity inherent in real-world scenarios. To overcome this, the authors propose a reinforcement learning–based alignment framework that treats the annotation distribution not as noise but as a valuable signal. The approach introduces a dynamic sample weighting mechanism that emphasizes highly ambiguous instances during optimization, jointly enhancing distribution matching and classification performance. Evaluated on benchmarks such as ChaosNLI, the method achieves substantial improvements, reducing the Jensen–Shannon divergence by up to 62.1% and increasing F1 scores by as much as 16.7%, thereby significantly improving the model’s capacity to reflect the diversity of human judgments.

Abstract

Many human-centered tasks, including natural language inference (NLI) and emotion recognition (ER), have multiple plausible interpretations, leading to label ambiguity and challenging disagreements across human annotators. As LLMs are increasingly deployed in real-world settings, faithfully modeling such ambiguity is essential to identify contested inputs, preserve variability in ambiguous cases, and capture the full distribution of human judgments. Yet, existing LLM alignment approaches have predominantly assumed a single correct label, excluding annotator disagreement during optimization. Instead of treating this ambiguity as noise, we show how to treat it as information that improves model behavior through a new algorithm called SMARTLY HANDLING AMBIGUOUS LABELS IN ALIGNING LLMS (SHALA-LLM). This reinforcement learning framework provides a new way for LLMs to learn directly from annotator distributions while dynamically prioritizing highly ambiguous samples during optimization. Experiments on ambiguity-sensitive NLI and ER benchmarks, including ChaosNLI, GoEmotions, and MSP-Podcast, demonstrate that SHALA-LLM improves agreement with annotator label distributions, e.g. on ChaosNLI, it reduces Jensen-Shannon Distance by up to 62.1%. At the same time, SHALA-LLM improves F1 by up to 16.7%, showing that modeling annotator disagreement can also strengthen classification performance.

Problem

• label ambiguity
• LLM alignment
• annotator disagreement
• natural language inference
• emotion recognition

Innovation

• label ambiguity
• LLM alignment
• reinforcement learning
• annotator disagreement
• distribution modeling