Information-Based Exploration via Random Features for Reinforcement Learning

2026-07-20

AI Summary

This work addresses the lack of theoretical guarantees for efficient exploration in deep reinforcement learning with uncountable state spaces. It proposes Random Feature Information Gain (RFIG), a novel method that, for the first time, integrates Bayesian kernel methods with random Fourier features to explicitly approximate information gain and construct an optimistic exploration bonus with provable error bounds. By circumventing the black-box nature of uncertainty estimation in neural networks, RFIG offers a transparent and theoretically grounded approach that seamlessly integrates into mainstream deep reinforcement learning algorithms. Empirical evaluations demonstrate that RFIG achieves performance on par with state-of-the-art exploration methods across a range of control and navigation tasks, while providing stronger theoretical interpretability.

Abstract

Representation learning has enabled classical exploration strategies to be extended to deep Reinforcement Learning (RL), but often makes algorithms more complex and theoretical guarantees harder to establish. We introduce Random Feature Information Gain (RFIG), grounded in Bayesian kernel methods theory, which uses random Fourier features to approximate information gain and compute exploration bonuses in non-countable spaces. We provide error bounds on information gain approximation and avoid the black-box aspects of neural network-based uncertainty estimation, for optimism-based exploration. We present practical details that make RFIG scalable to deep RL scenarios, enabling smooth integration into standard deep RL algorithms. Experimental evaluation across diverse control and navigation tasks demonstrates that RFIG achieves competitive performance with well-established deep exploration methods while offering superior theoretical interpretation.

Problem

• information gain
• exploration
• reinforcement learning
• random features
• Bayesian kernel methods

Innovation

• Random Fourier Features
• Information Gain
• Bayesian Kernel Methods
• Optimistic Exploration
• Representation Learning