Risk-Aware Preference Learning for Stochastic Outcomes

2026-07-16

AI Summary

This work addresses a critical limitation in existing preference learning methods, which typically rely on expected utility theory and thereby overlook humans’ systematic risk-sensitive behaviors toward rare but high-consequence events—such as robot collisions—under uncertainty. To bridge this gap, the paper introduces cumulative prospect theory (CPT) into the Bradley–Terry preference learning framework for the first time, explicitly modeling human tendencies to overweight extreme negative outcomes and exhibit loss aversion. The resulting risk-aware reward learning algorithm better captures human preferences in safety-critical scenarios. Experimental results demonstrate that, in tasks simulating risk-sensitive user preferences, the proposed CPT-based approach significantly outperforms conventional expected utility models, achieving lower regret and higher alignment accuracy between the learned reward function and true human preferences.

Abstract

Learning reward functions from human preferences is a widely used approach for aligning robot behavior with user expectations in human-robot interaction. Most existing approaches assume that humans evaluate uncertain outcomes using expected utility (EU), aggregating outcome utilities linearly with their probabilities. However, behavioral evidence shows that humans are systematically risk-sensitive, overweighting rare negative events and exhibiting loss aversion. We study the consequences of this mismatch in social robot navigation, where safety-critical outcomes (e.g., collisions) are rare but highly consequential. We compare EU with Cumulative Prospect Theory (CPT), a nonlinear model of human decision-making, within a Bradley-Terry preference learning framework. Our preliminary experiments show that when preferences are generated by risk-sensitive users, CPT-based learners recover reward functions with substantially lower regret compared to EU-based learners. Our results highlight the importance of modeling human risk sensitivity when learning rewards from preferences over stochastic robot outcomes.

Problem

• risk sensitivity
• preference learning
• stochastic outcomes
• reward learning
• human-robot interaction

Innovation

• risk-sensitive preference learning
• Cumulative Prospect Theory
• reward learning
• human-robot interaction
• stochastic outcomes