Language-Guided Multi-Agent Learning in Simulations: A Unified Framework and Evaluation

2025-06-01

AI Summary

This work addresses the challenge of effectively integrating large language models (LLMs) into multi-agent reinforcement learning (MARL) to enhance agent coordination, symbolic communication, and zero-shot generalization in simulated game environments. We propose LLM-MARL, a novel framework featuring a tripartite “Coordinator–Communicator–Memory” architecture that enables dynamic subgoal generation, symbolic cross-agent communication, and episodic memory retrieval, supporting end-to-end, language-guided MARL training. The method integrates PPO with language-conditioned loss and an LLM query gating mechanism, and is compatible with Google Research Football, MAgent Battle, and StarCraft II. Experiments demonstrate significant improvements over MAPPO and QMIX in win rate, coordination score, and zero-shot generalization. Ablation studies confirm that subgoal generation and language-based communication each contribute over 35% performance gain. Emergent behaviors—including role specialization and communication-driven tactics—are also observed.

Technology Category

• Multiagent Systems: Multiagent Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This paper introduces LLM-MARL, a unified framework that incorporates large language models (LLMs) into multi-agent reinforcement learning (MARL) to enhance coordination, communication, and generalization in simulated game environments. The framework features three modular components of Coordinator, Communicator, and Memory, which dynamically generate subgoals, facilitate symbolic inter-agent messaging, and support episodic recall. Training combines PPO with a language-conditioned loss and LLM query gating. LLM-MARL is evaluated in Google Research Football, MAgent Battle, and StarCraft II. Results show consistent improvements over MAPPO and QMIX in win rate, coordination score, and zero-shot generalization. Ablation studies demonstrate that subgoal generation and language-based messaging each contribute significantly to performance gains. Qualitative analysis reveals emergent behaviors such as role specialization and communication-driven tactics. By bridging language modeling and policy learning, this work contributes to the design of intelligent, cooperative agents in interactive simulations. It offers a path forward for leveraging LLMs in multi-agent systems used for training, games, and human-AI collaboration.

Problem

• Enhancing multi-agent coordination via LLM integration
• Improving communication through symbolic messaging in simulations
• Boosting zero-shot generalization in game environments

Innovation

• LLM-MARL integrates LLMs with MARL
• Modular Coordinator, Communicator, Memory components
• PPO training with language-conditioned loss