🤖 AI Summary
Decentralized Multi-Agent Reinforcement Learning (DMARL) faces fundamental challenges in ensuring policy compatibility, coping with communication constraints, and preserving privacy. Method: This paper proposes a temporally causal symbolic knowledge-guided distributed training framework. It pioneers the extension of formal policy compatibility verification to a symbolic knowledge space endowed with temporal-causal structure, integrating Symbolic AI, temporal logic modeling, multi-agent RL, and formal verification techniques. Contribution/Results: The framework enables scalable, decentralized training with theoretical guarantees—without requiring global state information or frequent inter-agent communication. It significantly improves policy compatibility and sample efficiency. Empirical evaluation on collaborative robot–drone tasks demonstrates accelerated convergence and higher task success rates.
📝 Abstract
Reinforcement learning (RL) algorithms can find an optimal policy for a single agent to accomplish a particular task. However, many real-world problems require multiple agents to collaborate in order to achieve a common goal. For example, a robot executing a task in a warehouse may require the assistance of a drone to retrieve items from high shelves. In Decentralized Multi-Agent RL (DMARL), agents learn independently and then combine their policies at execution time, but often must satisfy constraints on compatibility of local policies to ensure that they can achieve the global task when combined. In this paper, we study how providing high-level symbolic knowledge to agents can help address unique challenges of this setting, such as privacy constraints, communication limitations, and performance concerns. In particular, we extend the formal tools used to check the compatibility of local policies with the team task, making decentralized training with theoretical guarantees usable in more scenarios. Furthermore, we empirically demonstrate that symbolic knowledge about the temporal evolution of events in the environment can significantly expedite the learning process in DMARL.