Sparse Gaussian-Mixture-Model Q-Functions via Hadamard Overparametrization for Online Reinforcement Learning

2026-07-26

AI Summary

This work addresses the challenges of streaming non-stationary data, distributional shifts, and the trade-off between model interpretability and parameter efficiency in online reinforcement learning. To this end, we propose an online policy iteration framework that employs a sparse Gaussian mixture model to represent the Q-function. Our approach integrates experience replay with online gradient optimization on a Riemannian manifold and introduces an interpretable sparse structure via Hadamard over-parameterization, allowing the geometric roles of individual Gaussian components in the state-action space to emerge naturally. Experimental results demonstrate that the proposed method achieves performance comparable to or better than deep reinforcement learning baselines while significantly reducing the number of parameters, and it maintains strong generalization and higher sample efficiency even under low-parameter regimes.

Abstract

This paper develops an online, off-policy policy-iteration framework for reinforcement learning (RL), based on sparse Gaussian-mixture-model Q-functions (S-GMM-QFs). The framework reconciles streaming, non-stationary data with the Riemannian structure of the parameter space while handling distributional mismatch through experience replay. S-GMM-QFs are introduced via Hadamard overparametrization, enabling interpretable sparsification through smooth regularization that facilitates Riemannian-based optimization. Overparametrization allows the framework to adaptively identify meaningful components from a large initial pool, yielding sparse models where interpretability emerges naturally from geometry: each component's parameters (means and covariances) explicitly encode its geometric role in the ambient state-action space. These geometric roles are learned through online gradient descent on a smooth objective over a (Cartesian-product) Riemannian manifold. Numerical tests demonstrate that S-GMM-QFs match or exceed deep RL methods while using substantially fewer parameters and achieving faster improvement per observed transition. Notably, parameter efficiency and interpretability combine to maintain strong generalization in low-parameter regimes where sparsified deep RL approaches degrade.

Problem

• online reinforcement learning
• distributional mismatch
• parameter efficiency
• model interpretability
• non-stationary data

Innovation

• Hadamard overparametrization
• Sparse Gaussian mixture model
• Riemannian optimization
• Interpretable reinforcement learning
• Online policy iteration