Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

2025-07-02

AI Summary

This study addresses the challenge of aligning large language models (LLMs) with diverse and potentially conflicting human preferences—i.e., achieving multi-objective alignment. We formulate value alignment as a multi-objective optimization problem and propose the Gradient-Adaptive Pareto Optimization (GAPO) framework, along with its personalized extension, P-GAPO. GAPO is the first method to integrate multi-gradient descent with adaptive gradient scaling into LLM alignment, ensuring sustained convergence to Pareto-optimal solutions. The approach unifies reinforcement learning from human feedback (RLHF), multi-objective gradient coordination, and theory-grounded Pareto analysis. Experiments on Mistral-7B demonstrate that GAPO significantly outperforms existing state-of-the-art methods across two core dimensions—helpfulness and harmlessness—validating its effectiveness in multi-objective co-optimization and its generalizability to user-customized alignment.

Technology Category

• Humans and AI: Learning Human Values and Preferences
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reinforcement Learning from Human Feedback (RLHF) has emerged as a powerful technique for aligning large language models (LLMs) with human preferences. However, effectively aligning LLMs with diverse human preferences remains a significant challenge, particularly when they are conflict. To address this issue, we frame human value alignment as a multi-objective optimization problem, aiming to maximize a set of potentially conflicting objectives. We introduce Gradient-Adaptive Policy Optimization (GAPO), a novel fine-tuning paradigm that employs multiple-gradient descent to align LLMs with diverse preference distributions. GAPO adaptively rescales the gradients for each objective to determine an update direction that optimally balances the trade-offs between objectives. Additionally, we introduce P-GAPO, which incorporates user preferences across different objectives and achieves Pareto solutions that better align with the user's specific needs. Our theoretical analysis demonstrates that GAPO converges towards a Pareto optimal solution for multiple objectives. Empirical results on Mistral-7B show that GAPO outperforms current state-of-the-art methods, achieving superior performance in both helpfulness and harmlessness.

Problem

• Aligning LLMs with diverse human preferences
• Balancing conflicting objectives in multi-optimization
• Adapting gradients for optimal trade-offs between objectives

Innovation

• Gradient-Adaptive Policy Optimization for multi-objective alignment
• Adaptive gradient rescaling to balance conflicting objectives
• P-GAPO incorporates user preferences for Pareto solutions