ERPPO: Entropy Regularization-based Proximal Policy Optimization

2026-05-13

AI Summary

This work addresses the challenge of policy convergence and target localization accuracy in multi-agent reinforcement learning under non-stationary, high-dimensional observations, where MAPPO struggles due to observation-induced non-stationarity. The authors propose ERPPO, a novel approach that uniquely integrates dynamic entropy regularization with estimation of observational ambiguity. Specifically, a Distributional Spatio-Temporal Ambiguity (DSA) learner quantifies environmental uncertainty, enabling adaptive switching between L1 and L2 regularization: L1 promotes exploration in high-ambiguity regions, while L2 stabilizes optimization in low-ambiguity regions. Evaluated in an AirSim maritime search-and-rescue simulation, ERPPO significantly improves target localization accuracy, effectively suppresses false detections under visual uncertainty, and achieves more efficient policy gradient updates.

Abstract

Multi-Agent Proximal Policy Optimization (MAPPO) is a variant of the Proximal Policy Optimization (PPO) algorithm, specifically tailored for multi-agent reinforcement learning (MARL). MAPPO optimizes cooperative multi-agent settings by employing a centralized critic with decentralized actors. However, in case of multi-dimensional environment, MAPPO can not extract optimal policy due to non-stationary agent observation. To overcome this problem, we introduce a novel approach, Entropy Regularization-based Proximal Policy Optimization (ERPPO). For the policy optimization, we first define the object detection ambiguity under multi-dimensional observation environment. Distributional Spatiotemporal Ambiguity (DSA) learner is trained to estimate object detection uncertainty in non-stationary constraints. Then, we enhance PPO with a novel Entropy Regularization term. This regularization dynamically adjusts the policy update by applying a stronger (L1) regularization in high-ambiguity observation to encourage significant exploratory actions and a weaker (L2) regularization in low-ambiguity observation to stabilize the proximal policy optimization. This approach is designed to enhance the probability of successful object localization in time-critical operations by reducing detection failures and optimizing search policy. Experiments on a testbed with AirSim-based maritime searching scenarios show that the proposed ERPPO improves accuracy performance. Our proposed method improves higher gradient than MAPPO. Qualitative results confirm that ERPPO effectiveness in terms of suppressing false detection in visually uncertain conditions.

Problem

• multi-agent reinforcement learning
• non-stationary observation
• object detection ambiguity
• policy optimization
• entropy regularization

Innovation

• Entropy Regularization
• Proximal Policy Optimization
• Multi-Agent Reinforcement Learning
• Distributional Spatiotemporal Ambiguity
• Adaptive Regularization