ODYSSE: Episode-wise Policy Optimization for Personalized Agentic Reasoning

2026-07-28

AI Summary

This work addresses the challenge of personalized agent reasoning under ambiguous user requests and open-ended solution spaces by introducing the ODYSSE framework, which leverages reinforcement fine-tuning to dynamically understand user preferences and deliver tailored services. The core innovation lies in the Episode-wise Group Relative Policy Optimization (ESPO) algorithm, which transcends conventional step-level policy updates by optimizing over entire interaction episodes. ESPO integrates episode-level rewards, advantage estimation, and a unified batch action sampling mechanism to effectively model long-horizon, strongly dependent reasoning processes. Experimental results demonstrate that the proposed method significantly outperforms both specialized and general-purpose LVLMs on real-world, long-horizon GUI personalization tasks, validating its efficacy in personalized agent reasoning.

Abstract

Agentic systems have rapidly advanced in their ability to interact with real-world environments, leverage external tools, and provide services for users. However, unlike natural-world tasks that assume well-defined instructions, human-centered scenarios are characterized by ambiguous requests that lead to large, open-ended solution spaces. Decoding users' personalized preferences is therefore essential for narrowing the candidate solution space. This introduces a new challenge, personalized agentic reasoning, which requires agents to jointly interact with both users and environments to deliver personalized services. In this paper, we present ODYSSE, a Reinforced Fine-Tuning (RFT) framework for personalized agentic reasoning. At its core, ODYSSE proposes Episode-wise GRPO (ESPO), a novel extension of Group Relative Policy Optimization (GRPO) designed to address long action horizons and strong cross-step dependencies in personalized agentic reasoning. Rather than optimizing individual steps independently, ESPO introduces an episode-level reward mechanism together with episodic advantage estimation, enabling upstream evidence to effectively guide downstream personalized decisions and allowing agents to progressively resolve ambiguous user requests across multiple interaction steps. We further propose an episodic batch sampler that groups actions from the same episode into unified training batches, facilitating coherent optimization under ESPO. We evaluate ODYSSE on realistic long-horizon personalized GUI reasoning tasks. Experimental results demonstrate that ODYSSE consistently outperforms both specialist and general-purpose LVLMs, highlighting its effectiveness for personalized agentic reasoning.

Problem

• personalized agentic reasoning
• ambiguous user requests
• open-ended solution spaces
• user preference decoding
• long-horizon interaction

Innovation

• personalized agentic reasoning
• Episode-wise GRPO
• Reinforced Fine-Tuning
• episodic advantage estimation
• long-horizon reasoning