PolyAlign: Conditional Human-Distribution Alignment

2026-06-11

AI Summary

This work addresses the limitation of conventional alignment methods that optimize language models toward a single, monolithic assistant behavior, thereby neglecting the natural diversity of human responses across linguistic styles, tasks, and conversational contexts. To overcome this, the authors propose PolyAlign, a conditional human distribution alignment framework that explicitly models the distribution of human responses under distinct contextual conditions—such as language, interaction type, response family, and length—by partitioning data into contextual buckets. PolyAlign integrates bucket-aware supervised fine-tuning (Bucket-Aware SFT) with Human Distribution Preference Optimization (HDPO), a preference-based method regularized by distributional distance metrics. This approach achieves, for the first time, interaction-aware multi-distribution alignment, significantly enhancing conditional naturalness and distributional fidelity of model responses in both Chinese and English, across single- and multi-turn dialogues, while preserving strong task utility.

Abstract

Post-training methods such as supervised fine-tuning (SFT) and preference optimization typically align language models toward a single global assistant behavior. While effective for improving average helpfulness, this can suppress the natural variation of human responses across languages, tasks, and dialogue settings. We study this problem as conditional human-distribution alignment: models should match the human response distribution appropriate to the current interaction context, rather than a universal response style. We introduce PolyAlign, a distribution-aware alignment framework that organizes bilingual interaction data into bucket-specific human reference distributions defined by language, interaction track, response family, and length. PolyAlign combines Bucket-Aware SFT, which balances optimization across heterogeneous buckets, with Human-Distribution Preference Optimization (HDPO), which regularizes preference learning using critic-estimated distance to bucket-specific human support. Across a bilingual evaluation suite covering English and Chinese single- and multi-turn settings, PolyAlign improves conditional naturalness and distributional faithfulness while preserving competitive task utility. The results suggest that post-training should move beyond global alignment objectives toward interaction-aware alignment with human response distributions.

Problem

• conditional alignment
• human response distribution
• post-training
• language models
• distributional faithfulness

Innovation

• conditional alignment
• human response distribution
• distribution-aware optimization
• bilingual alignment
• preference optimization