Problem
Research questions and friction points this paper is trying to address.
Zero-Sum Markov Games
Last-Iterate Convergence
Bandit Feedback
Independent Learning
Fast Last-Iterate Convergence in Zero-Sum Markov Games with Bandit Feedback
🤖 AI Summary
This study addresses the slow last-iterate convergence of independent learning strategies under bandit feedback in unknown zero-sum Markov games. To this end, we propose an adaptive regularized temporal difference (TD) learning algorithm. By decoupling value estimation from policy updates and integrating logarithmic barrier regularization with TD averaging, the proposed method achieves rapid and stable convergence without requiring prior knowledge of the time horizon or confidence parameters. Theoretical analysis demonstrates that, under identical assumptions, the algorithm attains a high-probability convergence bound of O(t^{-1/4}) on the duality gap, significantly improving upon the best existing theoretical results. This work provides a more efficient solution paradigm for online game-theoretic learning in multi-agent systems.
📝 Abstract
We study last-iterate convergence in unknown two-player zero-sum discounted Markov games with bandit feedback. The players learn independently along a single trajectory without observing each other's actions. We develop Adaptive Regularized TD Learning (ARTD), which achieves a $\widetilde{\mathcal{O}}(t^{-1/4})$ duality gap bound for the current policies under a uniform hitting time assumption, with high probability simultaneously over all rounds and starting states. This improves the $\widetilde{\mathcal{O}}(t^{-1/(9+\nu)})$ rate of Cai et al. (2023), for any fixed $\nu>0$, under the same feedback model and hitting time assumption. Our algorithm requires no knowledge of the hitting time bound, the time horizon, or the confidence level. To stabilize policy learning as value estimates change, we separate fast temporal difference averaging from bounded value updates. We adapt log-barrier regularization to the progress of value estimation, controlling both policy and value errors throughout learning. Together, these mechanisms enable fast convergence of the policies actually played, even when the players learn independently from bandit feedback.
🔎 Similar Papers
