🤖 AI Summary
Existing multi-agent reinforcement learning (MARL) methods predominantly focus on individual policy optimization while neglecting inter-agent interaction, leading to policy homogenization and poor adaptability. To address this, we propose CoDiCon—a novel MARL framework that, for the first time, integrates the sociological concept of constructive conflict into MARL. CoDiCon introduces a centralized intrinsic reward mechanism grounded in rank-based features, implemented via a parameterized reward module and a bilevel optimization architecture to dynamically balance competition and cooperation: competitive intrinsic rewards stimulate policy exchange and enhance diversity, while centralized rank-based features ensure collaborative consistency. Evaluated on the SMAC and Google Research Football (GRF) benchmarks, CoDiCon significantly outperforms state-of-the-art methods, demonstrating synergistic improvements in policy diversity, team adaptability, and task performance.
📝 Abstract
In recent years, diversity has emerged as a useful mechanism to enhance the efficiency of multi-agent reinforcement learning (MARL). However, existing methods predominantly focus on designing policies based on individual agent characteristics, often neglecting the interplay and mutual influence among agents during policy formation. To address this gap, we propose Competitive Diversity through Constructive Conflict (CoDiCon), a novel approach that incorporates competitive incentives into cooperative scenarios to encourage policy exchange and foster strategic diversity among agents. Drawing inspiration from sociological research, which highlights the benefits of moderate competition and constructive conflict in group decision-making, we design an intrinsic reward mechanism using ranking features to introduce competitive motivations. A centralized intrinsic reward module generates and distributes varying reward values to agents, ensuring an effective balance between competition and cooperation. By optimizing the parameterized centralized reward module to maximize environmental rewards, we reformulate the constrained bilevel optimization problem to align with the original task objectives. We evaluate our algorithm against state-of-the-art methods in the SMAC and GRF environments. Experimental results demonstrate that CoDiCon achieves superior performance, with competitive intrinsic rewards effectively promoting diverse and adaptive strategies among cooperative agents.