Agentic Multi-Turn Reasoning: A Fairness Approach

2026-09-27

AI Summary

This study addresses the long-horizon credit assignment challenge caused by sparse supervision signals in multi-turn reasoning of LLM agents, as well as the suppression of critical rare reasoning paths due to data imbalance. To tackle these issues, this work proposes Fair-MPO, a fair multi-level preference optimization framework. This method pioneers a novel paradigm that integrates multi-level preference optimization with fairness objectives, incorporating a multi-turn planning toolchain, verification mechanisms, and memory update techniques to effectively mitigate dominant mode bias while preserving computational efficiency. Experimental results demonstrate that Fair-MPO achieves state-of-the-art performance across multiple agent reasoning benchmarks, significantly enhancing decision-making accuracy and robustness in complex scenarios.

Abstract

Recent advances in Large Language Models (LLMs) have enabled agentic systems capable of solving complex tasks through multi-turn planning, tool use, verification, and memory updates. However, learning agentic systems remains difficult due to two fundamental challenges, i.e., (1) long-horizon credit assignment, where supervision is available only at the final outcome, and (2) imbalanced data distributions, where dominant data patterns bias optimization and weaken adaptation to rare but informative reasoning behaviors. In this paper, we propose Fair Multi-Level Preference Optimization (Fair-MPO or $\Phi$-MPO), a new preference optimization framework for agentic learning. We first show that Multi-Level Preference Optimization provides a principled and more computationally efficient framework for long-horizon reasoning. Then, we introduce a Fair Multi-Level Objective that addresses imbalance in agentic learning. We provide a comprehensive theoretical analysis demonstrating that our approach addresses both long-horizon reasoning and data imbalance. Our experiments on agentic reasoning benchmarks demonstrate that our approach achieves State-of-the-Art (SOTA) performance.

Problem

• Agentic Systems
• Long-horizon Credit Assignment
• Data Imbalance
• Multi-Turn Reasoning
• Preference Optimization

Innovation

• Agentic Multi-Turn Reasoning
• Preference Optimization
• Long-Horizon Credit Assignment
• Data Imbalance
• Fair Multi-Level Objective