Adjoint Guidance Flow: Amortized Critic Guidance for VLA Policies

2026-09-28

AI Summary

This study addresses the limitations of Vision-Language-Action (VLA) policies in optimizing long-term rewards and the substantial computational overhead associated with conventional critic-guided methods. To this end, we propose a lightweight, gradient-free, trajectory-aware critic guidance approach that formulates critic guidance as an optimal control problem. By integrating flow models with the adjoint method, our approach achieves trajectory-aware guidance through adjoint network amortization while keeping VLA parameters frozen throughout the process. As the first implementation in this direction, the proposed method significantly enhances performance across multiple benchmarks. Compared to QGF, it achieves a 3.6× increase in inference speed and a 7× reduction in parameter count. Furthermore, the approach demonstrates strong robustness, substantially lowering both training and inference costs.

Abstract

Flow-based Vision-Language-Action (VLA) policies are typically trained by behavior cloning and thus do not explicitly optimize long-term task return. Critic guidance steers generation toward higher-value actions, but existing methods differentiate the critic through a one-step surrogate of the sampler and back-propagate a critic ensemble at every flow step. In contrast, here we propose Adjoint Guidance Flow (AGF), which amortizes trajectory-aware critic guidance into a lightweight guidance network while preserving the pretrained VLA policy. Specifically, we formulate critic-guided flow generation as a deterministic optimal control problem, whose optimal guidance is a costate that carries the terminal critic gradient back through the remaining flow, and regress the guidance network onto this costate while keeping both the VLA and critic frozen. This design provides favorable memory and throughput scaling during training, and inference needs one guidance-network forward pass per step, without the critic ensemble, back-propagation, or adjoint computation. Across LIBERO, RoboCasa, and LIBERO-Pro, AGF consistently improves pretrained VLAs, remains competitive with critic-guidance and policy-fine-tuning baselines, and is the most robust method when a single guidance strength is deployed across tasks. Compared with QGF, AGF runs $3.6\times$ faster per guidance step with $7.0\times$ fewer parameters, with comparable and even better performance, showing that critic guidance can be trajectory-aware and lightweight.

Problem

• Vision-Language-Action policies
• Critic guidance
• Long-term task return
• Behavioral cloning
• Computational efficiency

Innovation

• Adjoint Guidance Flow
• Vision-Language-Action
• Optimal Control
• Amortized Critic Guidance
• Flow Matching