Towards Multi-agent Reinforcement Learning based Traffic Signal Control through Spatio-temporal Hypergraphs

2024-04-17

IEEE Transactions on Mobile Computing

AI Summary

Traditional traffic signal control neglects high-order spatiotemporal dependencies among intersections, hindering real-time, network-wide optimization. To address this, we propose a multi-agent traffic signal control system for urban road networks: (1) an edge-cooperative perception architecture enables dynamic, multi-intersection data acquisition; (2) hypergraph learning is innovatively embedded into the critic network of a multi-agent soft Actor-Critic (MA-SAC) framework to explicitly model high-order spatiotemporal interdependencies among intersections; and (3) a spatiotemporal hypergraph encoder jointly encodes dynamic topology and temporal evolution. Evaluated on multiple real-world and simulated road network datasets, our method achieves up to a 32.7% reduction in average vehicle travel time, alongside significant improvements in throughput and system stability. The source code and training environment are publicly released.

Technology Category

• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems
• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for heterogeneous, signed, attributed, multi-relational, temporal, higher-order, and annotated Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Traffic signal control systems (TSCSs) are integral to intelligent traffic management, fostering efficient vehicle flow. Traditional approaches often simplify road networks into standard graphs, which results in a failure to consider the dynamic nature of traffic data at neighboring intersections, thereby neglecting higher-order interconnections necessary for real-time control. To address this, we propose a novel TSCS framework to realize intelligent traffic control. This framework collaborates with multiple neighboring edge computing servers to collect traffic information across the road network. To elevate the efficiency of traffic signal control, we have crafted a multi-agent soft actor-critic (MA-SAC) reinforcement learning algorithm. Within this algorithm, individual agents are deployed at each intersection with a mandate to optimize traffic flow across the road network collectively. Furthermore, we introduce hypergraph learning into the critic network of MA-SAC to enable the spatio-temporal interactions from multiple intersections in the road network. This method fuses hypergraph and spatio-temporal graph structures to encode traffic data and capture the complex spatio-temporal correlations between multiple intersections. Our empirical evaluation, tested on varied datasets, demonstrates the superiority of our framework in minimizing average vehicle travel times and sustaining high-throughput performance. This work facilitates the development of more intelligent urban traffic management solutions. We release the code to support the reproducibility of this work at https://github.com/Edun-Eyes/TSC

Problem

• Enhancing traffic signal control using multi-agent reinforcement learning
• Capturing dynamic spatio-temporal traffic correlations via hypergraph learning
• Optimizing intersection traffic flow collaboratively across road networks

Innovation

• Multi-agent soft actor-critic reinforcement learning algorithm
• Hypergraph learning in critic network
• Fusion of hypergraph and spatio-temporal structures