Collaborative Personalized Preference Alignment for LLMs under Data Deficiency

2026-10-05

AI Summary

This study addresses the challenges of gradient interference and initialization learning in personalized alignment of large language models under data scarcity, arising from heterogeneous user preferences and conflicting objectives. To this end, we propose APO, a novel method that introduces an approximate Pareto optimal framework to coordinate multi-objective optimization via grouped compatible updates and a controlled ascent mechanism. By integrating federated learning with few-shot meta-learning, APO iteratively refines shared initialization parameters and establishes sub-optimality bounds to theoretically guarantee adaptive performance. Experiments on the Fed-ChatbotPA and UltraFeedback datasets demonstrate that APO significantly outperforms existing baselines using only 20 local samples per user, validating its effectiveness for efficient personalized alignment.

Abstract

Real-world users often exhibit highly heterogeneous preferences over multiple objectives for LLM responses. A lightweight aligner can tailor these responses to individual preferences, but scarce user-specific feedback makes personalized training difficult. Learning shared initializations across users can support few-shot adaptation. However, heterogeneous preferences and competing objectives cause gradient conflicts across users and within each user, hindering effective initialization learning. This raises a central question: \textbf{how can we collaboratively learn aligner initializations that support few-shot adaptation to diverse user preferences?} To answer this question, we propose \textbf{A}pproximate \textbf{P}areto \textbf{O}ptimality (APO). We first group users whose updates are compatible, so that their information can be combined with less interference. Within each group, we combine gradient descent with controlled ascent to coordinate competing objectives and move towards preference-specific points on the Pareto front. This produces an initialization that is close to the optima of the users in the group. We then iteratively refine it using updates from few-shot local adaptation, making it more effective for personalization. Furthermore, we establish conditional suboptimality bounds for a one-local-step collaborative update and characterize how initialization error affects subsequent stochastic adaptation. Experiments on Fed-ChatbotPA and UltraFeedback show consistent improvements over existing methods using only 20 local examples.

Problem

• Personalized Preference Alignment
• Data Deficiency
• Few-shot Adaptation
• Gradient Conflict
• Large Language Models

Innovation

• Personalized Preference Alignment
• Approximate Pareto Optimality
• Few-shot Adaptation
• Gradient Conflict Resolution
• Collaborative Initialization