Federated Variational Preference Alignment with Gumbel-Softmax Prior for Personalized User Preferences

2026-05-29

AI Summary

This work addresses the challenge in federated learning where a single reward model struggles to accommodate diverse or even conflicting user preferences—such as helpfulness versus harmlessness—leading to ineffective personalized alignment. To overcome this, the authors propose FedVPA-GP, a novel framework that introduces variational preference learning into the federated setting for the first time. By leveraging variational inference, the method disentangles user preferences and employs a federated mixture prior to mitigate posterior collapse caused by sparse and heterogeneous local data. An orthogonal loss is further incorporated to explicitly separate preference prototypes in the latent space. This approach enables privacy-preserving, personalized preference disentanglement and dynamic switching, significantly outperforming monolithic baselines on the HH-RLHF dataset and successfully achieving separation and flexible adaptation of conflicting intents.

Abstract

Federated Learning (FL) offers a privacy-preserving pathway for aligning Large Language Models (LLMs); however, existing frameworks typically enforce a monolithic reward model, inevitably averaging out inherently conflicting user preferences (e.g., helpfulness vs. harmlessness). While Variational Preference Learning (VPL) offers a pathway to personalization, adapting it to decentralized settings presents a fundamental challenge: posterior collapse driven by severe local data scarcity and heterogeneity. In this paper, we propose Federated Variational Preference Alignment with Gumbel-Softmax Prior (FedVPA-GP), a framework designed to disentangle diverse preferences without compromising privacy. To stabilize variational inference, we introduce a Federated Mixture Prior that enables clients to leverage the aggregate population distribution as a dynamic prior. Furthermore, we incorporate an Orthogonal Loss that explicitly enforces the separation of preference prototypes in the latent space. Experiments on the HH-RLHF dataset demonstrate that FedVPA-GP significantly outperforms monolithic baselines, successfully disentangling conflicting user intents and enabling dynamic preference switching.

Problem

• Federated Learning
• Personalized User Preferences
• Variational Preference Learning
• Posterior Collapse
• Preference Alignment

Innovation

• Federated Learning
• Variational Preference Learning
• Gumbel-Softmax Prior
• Preference Disentanglement
• Orthogonal Loss