Inverse RL Helps Align AI by Imitating Humans

2026-07-27

AI Summary

This work addresses the challenge of extracting interpretable and reusable implicit reward signals from human demonstrations alone—without task-specific preference annotations—to align language models. The authors propose PARED, a method grounded in inverse reinforcement learning that employs a lightweight discriminator to distinguish expert demonstrations from policy-generated responses in a response-level feature space, thereby explicitly recovering an implicit reward function. PARED is the first approach to construct an explicit, interpretable reward model using only demonstration data and no preference labels, enabling both inference-time reranking and policy gradient optimization while flexibly accommodating diverse audience-specific alignment requirements. Experiments demonstrate that PARED improves baseline policy performance even under unsupervised training loss, with further gains achieved when combined with standard fine-tuning.

Abstract

Language model alignment aims to make model behavior reliably reflect desirable properties such as helpfulness, safety, and instruction following. Current approaches typically use supervised fine-tuning on demonstrations or reinforcement learning with rewards derived from verifiers or human feedback. These paradigms leave an important question underexplored: can demonstrations alone yield an implicit reward that can be inspected, reused, and optimized on-policy to align AI? Motivated by inverse reinforcement learning, we introduce Projected Alignment Reward Estimated from Demonstrations (PARED). PARED recovers the implicit reward underlying expert demonstrations as an explicit function over a small set of response-level features, learned by a lightweight discriminator that separates demonstrations from the policy's own samples in this feature space. Unlike a standard reward model, PARED requires no task-specific preference annotations: demonstrations provide the task-specific supervision, which can be augmented with AI feedback as additional dimensions of supervision. Through experiments involving inference-time reranking and adversarial on-policy RL, we show that the recovered reward improves a base policy without a supervised loss and yields further gains when optimized after standard supervised fine-tuning. Additionally, we demonstrate that PARED can be used for contextual alignment, in which a single policy can be tailored to the preferences of different audiences.

Problem

• inverse reinforcement learning
• AI alignment
• demonstrations
• implicit reward
• language models

Innovation

• Inverse Reinforcement Learning
• Reward Modeling
• Demonstration-based Alignment
• Contextual Alignment
• On-policy Optimization