🤖 AI Summary
This work addresses stochastic team games under unknown dynamics. Methodologically, it proposes the Logit-Q dynamics framework—the first to couple Logit-response dynamics with Q-learning within an auxiliary stage game—where Q-functions drive state-dependent payoffs to enable efficient equilibrium learning. Technically, it introduces a novel analytical approach combining fictitious static Q-estimation scenarios with asymptotic coupling to the true dynamic environment, integrated with slowly varying epoch scheduling and coupling-based convergence analysis. This yields the first convergence and rationality guarantees for non-fully controllable stochastic games. Theoretically, the algorithm converges to an approximately optimal team equilibrium, with quantifiable approximation error; exhibits rationality against pure stationary-strategy opponents; and retains convergence when stage payoffs form a potential game and state transitions are controlled by a single agent.
📝 Abstract
We present a new family of logit-Q dynamics for efficient learning in stochastic games by combining the log-linear learning (also known as logit dynamics) for the repeated play of normal-form games with Q-learning for unknown Markov decision processes within the auxiliary stage-game framework. In this framework, we view stochastic games as agents repeatedly playing some stage game associated with the current state of the underlying game while the agents' Q-functions determine the payoffs of these stage games. We show that the logit-Q dynamics presented reach (near) efficient equilibrium in stochastic teams with unknown dynamics and quantify the approximation error. We also show the rationality of the logit-Q dynamics against agents following pure stationary strategies and the convergence of the dynamics in stochastic games where the stage-payoffs induce potential games, yet only a single agent controls the state transitions beyond stochastic teams. The key idea is to approximate the dynamics with a fictional scenario where the Q-function estimates are stationary over epochs whose lengths grow at a sufficiently slow rate. We then couple the dynamics in the main and fictional scenarios to show that these two scenarios become more and more similar across epochs due to the vanishing step size and growing epoch lengths.