Hybrid Advantage Estimation with Unified Critic for VLM Agentic Reinforcement Learning

2026-07-26

AI Summary

This work addresses the challenge that existing vision-language agents struggle to jointly optimize token-level and turn-level objectives in multi-turn decision-making, leading to inaccurate credit assignment. To resolve this, the authors propose the HyGAE framework, which establishes a unified theoretical formulation for both levels for the first time. Central to this framework is a hybrid advantage function that enables a single critic to estimate values at both granularities. Integrated within an actor-critic architecture and combined with level-specific discount factors and end-to-end reinforcement learning, the approach enables joint optimization of the dual objectives. Evaluated across five benchmark environments, HyGAE achieves an average success rate of 91%, outperforming current methods by 10% and demonstrating the critical role of the proposed hybrid advantage function in enhancing decision consistency.

Abstract

Large Vision-Language Models (VLMs) now act as agents in interactive environments, where success requires coherent reasoning and decision-making across turns. Although end-to-end training in agentic environments can improve such multi-turn decision-making abilities, current methods mainly rely on either token-wise optimization over concatenated token trajectories or turn-wise optimization with uniform within-turn credit. In this work, we establish theoretical formulations for the two levels of optimization and derive a hybrid advantage that serves both objectives. Furthermore, with an appropriate choice of discount factor and learning target, we prove that a unified critic model can estimate values for both turn-wise and token-wise. As such, we propose HyGAE, an actor-critic framework that jointly optimizes token- and turn-level objectives with the hybrid advantage and unified critic. We conduct extensive evaluations of HyGAE across five multi-turn decision-making environments, where it achieves an average success rate of 91% and a significant improvement of 10% over other methods. Furthermore, we provide an in-depth analysis showing that the exact analytic form of the hybrid advantage and return is crucial for optimization. Project Page: https://wx-zhang.github.io/hygae-web/.

Problem

• Vision-Language Models
• Reinforcement Learning
• Multi-turn Decision-making
• Advantage Estimation
• Actor-Critic

Innovation

• Hybrid Advantage Estimation
• Unified Critic
• Vision-Language Models
• Multi-turn Decision Making
• Actor-Critic Framework