Behavioral Foundation Models for Quality Diversity

2026-09-28

AI Summary

This study addresses the inefficiency of high-dimensional policy search and performance collapse under sparse rewards by proposing the BFM-QD framework. This method leverages an offline-pretrained behavior foundation model to map high-dimensional policies into a compact latent space, where quality-diversity search is subsequently performed. Furthermore, it introduces a closed-form, gradient-free policy improvement operator that eliminates the need for critics and backpropagation, thereby synergizing dimensionality reduction with efficient optimization. Experimental results demonstrate that BFM-QD significantly outperforms existing methods on continuous control benchmarks, exhibiting exceptional robustness and search efficiency, particularly in environments characterized by sparse and deceptive rewards.

Abstract

Behavioral Foundation Models (BFMs) are an emerging paradigm in reinforcement learning, playing a role analogous to large language models in natural language processing: they have shown remarkable versatility, enabling zero-shot performance, fast imitation, and online adaptation, all by exploiting the structure of a latent space. In this work, we investigate whether the latent behavioral space induced by BFMs can serve as an effective search space to discover large repertoires of behaviorally diverse and high-performing policies through Quality-Diversity (QD) methods. While QD methods generally search directly in high-dimensional policy parameter space, in this paper, we present BFM-QD, a framework that performs QD search in the compact latent space of a BFM. We further show that the BFM-QD framework provides a closed-form, gradient-free policy improvement operator that approximates a policy gradient update, but requires no critic training and no backpropagation. Across continuous-control benchmarks spanning dense locomotion, sparse navigation, and contact-rich manipulation, BFM-QD consistently outperforms parameter-space baselines, with particularly stark gains in sparse and deceptive settings, where all tested parameter-space QD methods collapse to near-zero performance. These results show the effectiveness of the BFM-QD framework, benefiting from the synergy between dimensionality reduction of the search space and offline pretraining from diverse behavioral data. This positions BFMs as a general-purpose backbone for QD optimization, extending their utility beyond zero-shot task solving to the discovery of diverse behavioral repertoires.

Problem

• Behavioral Foundation Models
• Quality Diversity
• Reinforcement Learning
• Latent Space Search
• Policy Discovery

Innovation

• Behavioral Foundation Models
• Quality-Diversity
• Latent Space Search
• Gradient-free Policy Improvement
• Reinforcement Learning