Efficient Exploration for Iterative Nash Preference Optimization

2026-05-31

AI Summary

This work addresses the failure of conventional scalar-reward-based alignment methods when human preferences exhibit cyclic or intransitive behavior, and overcomes the lack of effective exploration and theoretical guarantees in existing iterative Nash preference optimization approaches. The paper proposes the first online iterative Nash preference optimization algorithm, which formulates alignment as a preference game under general preference models. By integrating explicit adversarial exploration with a KL-regularized minimax optimization framework, the method enables policy updates without explicitly estimating the preference model. It combines a direct policy optimization structure with a sublinear regret bound of $O(\sqrt{T})$, improvable to $O(\log T)$ under an ideal minimax oracle, and elucidates the role of the KL regularization parameter in shaping the regret bound and governing the computation–statistical trade-off. Experiments demonstrate significant superiority over current NLHF methods on Llama-3-8B-Instruct.

Abstract

Preference alignment is central to improving large language models, but standard reward-based formulations can be restrictive when human preferences are cyclic, non-transitive, or otherwise not representable by a scalar reward. Nash Learning from Human Feedback (NLHF) addresses this limitation by modeling alignment as a preference game and targeting a Nash equilibrium rather than a reward maximizer. However, the learning-theoretic foundations of scalable NLHF remain limited. Existing regret guarantees rely on oracle-based methods that estimate a general preference model and solve KL-regularized minimax problems, while iterative NLHF methods directly optimize policy-level preference losses and are easier to implement but lack regret guarantees. We study online iterative NLHF under general preference models and identify exploration as the key obstacle. First, we show that standard iterative NLHF can suffer an exponential dependence on the KL-regularization parameter, revealing that implicit exploration through policy updates is insufficient for controlling regret. Second, we propose an explicitly exploratory iterative NLHF algorithm that combines SFT-based regularization with adversarial policy exploration. The resulting method retains the direct policy optimization structure of iterative NLHF, avoids explicit preference model estimation, and achieves an $O(\sqrt{T})$ regret bound without an exponential dependence on the KL-regularization parameter. We show that the regret can be improved to $O(\log(T))$ with access to a minimax oracle, clarifying the computational-statistical tradeoff in learning general preference games. Finally, we instantiate our method for LLM fine-tuning and evaluate it on \texttt{Llama-3-8B-Instruct} across multiple benchmarks, where explicit exploration yields consistent improvements over existing NLHF baselines.

Problem

• preference alignment
• Nash equilibrium
• exploration
• regret minimization
• human feedback

Innovation

• Nash Learning from Human Feedback
• explicit exploration
• regret bound
• preference alignment
• iterative policy optimization