Search-Aided Joint Agent-Environment Reinforcement Learning for Robust Lifelong Multi-Agent Path Finding with Rotations

2026-08-06

AI Summary

This work addresses the challenges in lifelong multi-agent path finding (LMAPF) posed by the neglect of realistic motion constraints—such as in-place rotation—and the difficulty of coordination in high-density environments. To tackle these issues, we propose SJRL, a search-augmented joint reinforcement learning framework that, for the first time, models environmental policies as learnable edge costs in a graph and jointly optimizes them with agent neural policies. SJRL integrates Causal PIBT for single-step lookahead search and backward Dijkstra for global guidance, effectively handling complex kinematic constraints and stringent safety requirements. Experimental results demonstrate that SJRL significantly outperforms Causal-PIBT across multiple high-density maps and validates its practical efficacy in a mixed-reality warehouse scenario involving eight physical robots and 248 virtual agents.

Abstract

Lifelong Multi-Agent Path Finding (LMAPF) requires repeatedly planning collision-free paths for agents that continuously receive new goals upon reaching their current ones. While many learning-based planners have been proposed for LMAPF, most rely on oversimplified kinematic assumptions that may overlook motion constraints critical to real-world performance. In this work, we study a more realistic LMAPF model derived from many real-world automated warehouse systems, termed LMAPF-R2, which incorporates robust safety constraints and in-place rotation constraints. These constraints substantially increase coordination difficulty, particularly in highly constrained spaces. To address these challenges, we propose Search-Aided Joint Reinforcement Learning (SJRL). We first augment neural policies with Causal PIBT, a single-step search-based planner that resolves agents' collisions and propagates their intentions. We then introduce a unified RL formulation that jointly optimizes agent and environment policies, where the environment policy learns graph edge costs to provide global movement guidance via backward Dijkstra search. Experiments demonstrate that SJRL achieves significant improvements over the strong search-based planner, Causal-PIBT, across multiple high-density maps. We further validate SJRL in a challenging mixed-reality warehouse environment with 8 physical robots and 248 virtual robots.

Problem

• Lifelong Multi-Agent Path Finding
• rotation constraints
• safety constraints
• real-world automated warehouse
• collision-free planning

Innovation

• Search-Aided Reinforcement Learning
• Joint Agent-Environment Optimization
• Multi-Agent Path Finding with Rotations
• Causal PIBT
• Lifelong Multi-Agent Planning