Direct Multi-Turn Preference Optimization for Language Agents

2024-06-21

Conference on Empirical Methods in Natural Language Processing

AI Summary

This work addresses two key limitations of Direct Preference Optimization (DPO) in multi-turn dialogue agent tasks: (1) bias arising from the intractable partition function, and (2) modeling inaccuracies due to inconsistent trajectory lengths between preferred and dispreferred responses. To this end, we propose Distribution-Matching Preference Optimization (DMPO). Methodologically, DMPO reformulates the RL objective via distribution matching over state-action occupancy measures—replacing conventional policy constraints to ensure theoretical soundness—and incorporates trajectory-length normalization into the Bradley–Terry preference model to mitigate length-induced bias. Empirically, DMPO achieves significant improvements over existing DPO variants across three multi-turn dialogue agent benchmarks, demonstrating enhanced training stability, cross-task generalization, and optimization efficiency.

Technology Category

Machine Learning: Learning Preferences or RankingsSearch and Optimization: Sampling/Simulation-based SearchKnowledge Representation and Reasoning: Preferences

Application Category

User Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Adapting Large Language Models (LLMs) for agent tasks is critical in developing language agents. Direct Preference Optimization (DPO) is a promising technique for this adaptation with the alleviation of compounding errors, offering a means to directly optimize Reinforcement Learning (RL) objectives. However, applying DPO to multi-turn tasks presents challenges due to the inability to cancel the partition function. Overcoming this obstacle involves making the partition function independent of the current state and addressing length disparities between preferred and dis-preferred trajectories. In this light, we replace the policy constraint with the state-action occupancy measure constraint in the RL objective and add length normalization to the Bradley-Terry model, yielding a novel loss function named DMPO for multi-turn agent tasks with theoretical explanations. Extensive experiments on three multi-turn agent task datasets confirm the effectiveness and superiority of the DMPO loss.

Problem

• Optimizing multi-turn tasks
• Addressing partition function challenges
• Improving language agent performance

Innovation

• Direct Multi-Turn Preference Optimization
• state-action occupancy measure constraint
• length normalization in Bradley-Terry model