Foresight Residual RL for Long-Horizon Robot Manipulation with Vision-Language-Action Models

2026-07-17

AI Summary

This work addresses the poor performance of existing Vision-Language-Action (VLA) models in long-horizon, high-precision assembly tasks, where tightly coupled subtasks and uncontrolled terminal state quality hinder success. To overcome these limitations, the authors propose a look-ahead residual reinforcement learning framework that trains a residual policy on top of a frozen VLA base policy. A visual look-ahead predictor, trained offline, estimates the probability of future task success and serves as a reward modulation factor to explicitly optimize the downstream compatibility of subtask handover states. By combining sparse success rewards with backward look-ahead induction, the method enables efficient policy optimization in Isaac Gym. Evaluated on a three-stage wrench-nut assembly task, the approach achieves an end-to-end success rate of 85.6%, substantially outperforming standard residual RL (54.5%) and the original VLA baseline.

Abstract

Vision-Language-Action (VLA) policies offer strong general-purpose manipulation priors, but often fail on tight-tolerance, contact-rich assembly due to long-horizon credit assignment and subtask coupling: a state that is geometrically successful for the current skill can be brittle for downstream skills. We show this failure mode in residual reinforcement learning (RL) over a frozen VLA base policy: constant sparse success rewards improve each subtask in isolation yet yield little or no gain when skills are chained, because terminal state quality is uncontrolled. We propose Foresight Residual RL, which optimizes handoff quality by augmenting each subtask's sparse success reward with an offline-estimated foresight value -- the probability of future subtask success conditioned on the terminal state of the current subtask. Concretely, we (i) train a visual foresight predictor from images of terminal states of the base policy, labeled using downstream rollout statistics, and (ii) train residual policies via backward foresight induction, using the predictor output as a reward multiplier. On a three-phase wrench-based nut-tightening assembly task in Isaac Gym (grasp, move-insert, rotate), our method achieves 85.6% full-task success, outperforming standard subtask residual RL (54.5%) and VLA baselines, while leaving per-subtask success unchanged. These results highlight that improving long-horizon performance requires shaping which successful states are produced at each sub-task, not only whether success occurs.

Problem

• long-horizon manipulation
• Vision-Language-Action models
• subtask coupling
• credit assignment
• terminal state quality

Innovation

• Foresight Residual RL
• Vision-Language-Action Models
• Long-Horizon Manipulation
• Handoff Quality
• Offline Foresight Prediction