Towards Low-Resource Alignment to Diverse Perspectives with Sparse Feedback

2025-10-17

AI Summary

This study addresses the challenge of aligning language models with diverse human values in low-resource settings. Methodologically, it introduces an efficient, lightweight pluralistic alignment framework integrating sparse feedback learning, model steering, and pluralistic decoding—requiring only 50 annotated samples to explicitly model the distributional diversity of values, thereby relaxing the “single optimal response” assumption. Its core innovation lies in dynamic, value-aware control of the output distribution during decoding, jointly optimizing value sensitivity and viewpoint inclusivity. Experiments demonstrate significant reductions in false positive rates for hate speech and misinformation detection. On the GlobalOpinionQA benchmark, the model achieves markedly improved alignment between its outputs and empirically observed human value distributions, confirming strong adaptability and generalization capability on complex sociopolitical issues.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments

Abstract

As language models have a greater impact on society, it is important to ensure they are aligned to a diverse range of perspectives and are able to reflect nuance in human values. However, the most popular training paradigms for modern language models often assume there is one optimal answer for every query, leading to generic responses and poor alignment. In this work, we aim to enhance pluralistic alignment of language models in a low-resource setting with two methods: pluralistic decoding and model steering. We empirically demonstrate that model steering offers consistent improvement over zero-shot and few-shot baselines with only 50 annotated samples. Our proposed methods decrease false positives in several high-stakes tasks such as hate speech detection and misinformation detection, and improves the distributional alignment to human values in GlobalOpinionQA. We hope our work highlights the importance of diversity and how language models can be adapted to consider nuanced perspectives.

Problem

• Aligning language models to diverse human perspectives with limited data
• Reducing false positives in hate speech and misinformation detection
• Improving pluralistic alignment through model steering and sparse feedback

Innovation

• Pluralistic decoding enhances diverse perspective alignment
• Model steering improves performance with sparse annotated samples
• Methods reduce false positives in high-stakes detection tasks