PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning

2026-09-25

AI Summary

This study addresses the limitations of human demonstrations, which often lack local precision, and the difficulty of guiding policy learning under sparse feedback. To overcome these challenges, this work proposes an inverse reinforcement learning framework that integrates demonstration data with progress-based heuristic feedback. This approach pioneers the joint modeling of progress feedback and demonstrations, fine-tuning a vision-language model to achieve four-dimensional alignment for reward function optimization. Experimental results demonstrate that the proposed framework significantly outperforms existing baselines, substantially improving returns and task success rates using only 20% of labeled demonstrations. Furthermore, it effectively mitigates reward hacking, thereby achieving efficient and robust reward learning.

Abstract

Human demonstrations provide dense policy-level information but sometimes lack local precision. Human feedback presents accurate local critiques, but offers sparse evaluations rather than direct policy guidance. We propose Progress-Heuristicized Inverse Reinforcement Learning (PHIRL), a data-efficient framework that learns robust reward functions by jointly leveraging demonstrations and feedback. Specifically, we use progress, a feedback modality that describes cumulative task completion. PHIRL iteratively infers a reward function from demonstrations via inverse reinforcement learning, calculates the learned rewards over the progress-annotated demonstrations, and aligns the rewards with progress annotations over four dimensions. We evaluate PHIRL on real and simulated robot tasks, with additional exploration using a fine-tuned vision-language model to provide progress feedback. Results demonstrate that PHIRL significantly outperforms the baselines, achieving substantially higher environmental return rewards and task success with only twenty percent of demonstrations annotated. Analysis of reward-hacking scenarios demonstrates that PHIRL learned reward functions are reliable against exploitation.

Problem

• Inverse Reinforcement Learning
• Human Demonstrations
• Human Feedback
• Reward Function
• Reward Hacking

Innovation

• Inverse Reinforcement Learning
• Progress Feedback
• Reward Alignment
• Vision-Language Model
• Reward Hacking