🤖 AI Summary
This work addresses the computational intractability and observational uncertainty inherent in multi-agent reinforcement learning for partially observable stochastic games (POSGs). To mitigate these challenges, we propose a modeling paradigm grounded in inter-agent information sharing. By introducing abstractions of public information and imposing structured assumptions—such as joint observability and delayed sharing—we construct tractable approximations of POSGs. We establish, for the first time, a theoretical necessity of information sharing for overcoming the computational hardness of POSGs. Furthermore, we design a unified learning framework that jointly computes equilibrium policies and team-optimal solutions. Our algorithm achieves quasi-polynomial time and sample complexity. Notably, for cooperative POSGs, it yields the first rigorous statistical and computational complexity bounds for team-optimal solutions—achieving both statistical and computational quasi-efficiency.
📝 Abstract
We study provable multi-agent reinforcement learning (RL) in the general framework of partially observable stochastic games (POSGs). To circumvent the known hardness results and the use of computationally intractable oracles, we advocate leveraging the potential emph{information-sharing} among agents, a common practice in empirical multi-agent RL, and a standard model for multi-agent control systems with communications. We first establish several computational complexity results to justify the necessity of information-sharing, as well as the observability assumption that has enabled quasi-efficient single-agent RL with partial observations, for efficiently solving POSGs. {Inspired by the inefficiency of planning in the ground-truth model,} we then propose to further emph{approximate} the shared common information to construct an {approximate model} of the POSG, in which planning an approximate emph{equilibrium} (in terms of solving the original POSG) can be quasi-efficient, i.e., of quasi-polynomial-time, under the aforementioned assumptions. Furthermore, we develop a partially observable multi-agent RL algorithm that is emph{both} statistically and computationally quasi-efficient. {Finally, beyond equilibrium learning, we extend our algorithmic framework to finding the emph{team-optimal solution} in cooperative POSGs, i.e., decentralized partially observable Markov decision processes, a much more challenging goal. We establish concrete computational and sample complexities under several common structural assumptions of the model.} We hope our study could open up the possibilities of leveraging and even designing different emph{information structures}, a well-studied notion in control theory, for developing both sample- and computation-efficient partially observable multi-agent RL.