Learning to Reflect: Hierarchical Multi-Agent Reinforcement Learning for CSI-Free mmWave Beam-Focusing

2026-03-07

AI Summary

This work addresses the high overhead from channel state information (CSI) dependency and the curse of dimensionality in centralized optimization for reconfigurable intelligent surface (RIS)-assisted millimeter-wave systems. To overcome these challenges, the authors propose a CSI-free hierarchical multi-agent reinforcement learning framework that leverages user location information instead of CSI. The architecture employs a two-level controller: a high-level module for user-reflector assignment and a low-level module for focal point optimization, enabling efficient beam focusing through decentralized execution. Trained and evaluated in a ray-tracing environment using the MAPPO algorithm under the centralized training with decentralized execution (CTDE) paradigm, the proposed method achieves a 2.81–7.94 dB improvement in received signal strength indicator (RSSI) over centralized baselines. It further demonstrates robustness and scalability under doubled user density, up to 0.5-meter positioning errors, and varying RIS sizes.

Technology Category

• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Reconfigurable Intelligent Surfaces promise to transform wireless environments, yet practical deployment is hindered by the prohibitive overhead of Channel State Information (CSI) estimation and the dimensionality explosion inherent in centralized optimization. This paper proposes a Hierarchical Multi-Agent Reinforcement Learning (HMARL) framework for the control of mechanically reconfigurable reflective surfaces in millimeter-wave (mmWave) systems. We introduce a"CSI-free"paradigm that substitutes pilot-based channel estimation with readily available user localization data. To manage the massive combinatorial action space, the proposed architecture utilizes Multi-Agent Proximal Policy Optimization (MAPPO) under a Centralized Training with Decentralized Execution (CTDE) paradigm. The proposed architecture decomposes the control problem into two abstraction levels: a high-level controller for user-to-reflector allocation and decentralized low-level controllers for low-level focal point optimization. Comprehensive ray-tracing evaluations demonstrate that the framework achieves 2.81-7.94 dB RSSI improvements over centralized baselines, with the performance advantage widening as system complexity increases. Scalability analysis reveals that the system maintains sustained efficiency, exhibiting minimal per-user performance degradation and stable total power utilization even when user density doubles. Furthermore, robustness validation confirms the framework's viability across varying reflector aperture sizes (45-99 tiles) and demonstrates graceful performance degradation under localization errors up to 0.5 m. By eliminating CSI overhead while maintaining high-fidelity beam-focusing, this work establishes HMARL as a practical solution for intelligent mmWave environments.

Problem

• Reconfigurable Intelligent Surfaces
• Channel State Information
• mmWave beam-focusing
• dimensionality explosion
• CSI estimation overhead

Innovation

• Hierarchical Multi-Agent Reinforcement Learning
• CSI-Free
• Reconfigurable Intelligent Surfaces
• mmWave Beam-Focusing
• Centralized Training with Decentralized Execution