ARS: Agentic Reward System for Robot Learning

2026-09-28

AI Summary

This study addresses the challenge of distinguishing effective state transitions from unproductive actions in robotic progress reward modeling by proposing a training-free visual-language model (VLM) reasoning framework. The method introduces a multi-agent collaboration mechanism wherein sub-agents generate proposals and a primary agent performs verification, integrating structured reasoning with adaptive visual inspection for offline trajectory analysis to achieve precise progress estimation and substantially suppress spurious reward signals. Evaluations on standard benchmarks and real-world industrial assembly line experiments demonstrate that the proposed framework outperforms existing baselines, effectively facilitating high-quality policy learning over long-horizon tasks.

Abstract

Progress reward modeling is the problem of estimating how a robot's behavior changes task progress over time. Reliable estimation requires distinguishing meaningful state changes from failed attempts and task-irrelevant actions. We introduce the Agentic Reward System (ARS), an inference framework for progress reward modeling with general-purpose vision-language models (VLMs), without additional reward-model training. Given an offline trajectory and a task instruction, ARS uses adaptive visual inspection for both event proposal and verification. A subagent proposes a task-relevant event timeline, which a primary agent verifies and revises before estimating per-frame progress. ARS can incorporate optional terminal outcome labels and visual references to inform its judgments. It can also audit progress estimates from external reward models. We evaluate ARS with a 27B VLM on a controlled semantic-mismatch benchmark and downstream policy learning in simulation and on a real robot. The benchmark reveals that several evaluated reward baselines assign spurious progress to wrong-object manipulation even in simple pick-and-place scenes. ARS better suppresses these errors and outperforms these baselines in simulation policy learning. We further demonstrate that ARS supports long-horizon policy learning from mixed-quality offline experience on real-robot multi-screw fastening in a full-scale laboratory replica of an industrial washing-machine assembly line. These results suggest that structured inference and verification can improve the usefulness of general-purpose VLMs for robot reward modeling. Code is at https://github.com/midea-ai/ars

Problem

• progress reward modeling
• robot learning
• task progress estimation
• reward modeling

Innovation

• Agentic Reward System
• Progress Reward Modeling
• Vision-Language Models
• Multi-Agent Verification
• Robot Learning