🤖 AI Summary
This work addresses the adaptive challenges posed by non-stationary task dynamics and evolving goals in visual reinforcement learning. We introduce diffusion policies to this setting for the first time, proposing an iterative denoising-based diffusion policy framework that directly generates temporally coherent and context-aware action sequences from high-dimensional visual observations. Coupled with a lightweight visual encoder, the framework enables efficient online adaptation. Evaluated on non-stationary benchmarks—including Procgen and PointMaze—our method outperforms PPO and DQN, achieving +12.7%–28.3% gains in average and peak reward while reducing action variance by 36.5%, demonstrating robust adaptation to drifting dynamics and shifting task objectives. Our core contribution is the pioneering integration of diffusion models into non-stationary visual RL, unifying efficient exploration with stable online adaptation.
📝 Abstract
This paper investigates the application of Diffusion Policy in non-stationary, vision-based RL settings, specifically targeting environments where task dynamics and objectives evolve over time. Our work is grounded in practical challenges encountered in dynamic real-world scenarios such as robotics assembly lines and autonomous navigation, where agents must adapt control strategies from high-dimensional visual inputs. We apply Diffusion Policy -- which leverages iterative stochastic denoising to refine latent action representations-to benchmark environments including Procgen and PointMaze. Our experiments demonstrate that, despite increased computational demands, Diffusion Policy consistently outperforms standard RL methods such as PPO and DQN, achieving higher mean and maximum rewards with reduced variability. These findings underscore the approach's capability to generate coherent, contextually relevant action sequences in continuously shifting conditions, while also highlighting areas for further improvement in handling extreme non-stationarity.