Position: Good Embodied Reward Models Need Bad Behavior Data

2026-05-31

AI Summary

Existing embodied reward models struggle to accurately evaluate robot behaviors due to a critical absence of data depicting failures, suboptimal actions, or hazardous conduct—collectively termed “bad behaviors”—leading to poor alignment with human preferences. This work systematically identifies the lack of such negative examples as a primary cause of reward model failure and proposes augmenting training data with real or synthetically generated bad behaviors to enhance the model’s ability to recognize and penalize unsafe, inefficient, or exploitative actions. To this end, we introduce a curated bad-behavior dataset, a generation engine for synthesizing diverse negative examples, and a fine-grained evaluation benchmark. Experiments demonstrate that incorporating even a small amount of bad behavior data substantially improves alignment with human judgments, effectively mitigates over-rewarding, and reduces deployment risks.

Abstract

This position paper argues that to obtain reliable embodied reward models, the community must invest in ``bad'' robot data: failed, suboptimal, error-prone, and even hazardous behaviors. While reward models are central to any foundation model's lifecycle, today's embodied reward models are trained primarily on successful behaviors. We analyze three state-of-the-art embodied reward models and find that they systematically over-reward behaviors that real human evaluators would penalize, including unsafe interactions, poor execution, and shortcut strategies that only superficially satisfy tasks. We attribute these failures to a key data gap: the scarcity of negative embodied data which is costly to collect and often filtered out or withheld in existing robotics datasets. Furthermore, we show that even modest exposure to real bad behavior data can improve alignment with human preferences and reduce costly false positives. We therefore call on the embodied AI community to curate and release their bad robot data, build synthetic bad data generation engines, develop more decentralized physical evaluation systems, and design benchmarks for fine-grained embodied reward model evaluations.

Problem

• embodied reward models
• bad behavior data
• human preference alignment
• robotics datasets
• reward overestimation

Innovation

• embodied reward models
• bad behavior data
• human preference alignment
• negative data scarcity
• reward model evaluation