A Self-Evolving Default Action for Cooperative Tasks with Continuous Action Space

2026-07-20

AI Summary

This work addresses the challenge of biased policy gradients in existing counterfactual credit assignment methods when applied to multi-agent cooperative tasks with continuous action spaces, which often impedes convergence to local optima. To overcome this limitation, the paper introduces the SAFE framework, which for the first time naturally extends an unbiased counterfactual baseline to continuous action spaces. SAFE constructs the counterfactual baseline by sampling self-evolved default actions from an experience replay buffer, eliminating the need for additional simulations or prior knowledge while accurately evaluating each agent’s contribution. This approach guarantees unbiased deterministic policy gradients and provably converges to a local optimum. Empirical results on cooperative vehicle tasks demonstrate that SAFE significantly outperforms state-of-the-art methods, confirming its effectiveness and robustness.

Abstract

Counterfactual credit assignment has proven effective in multi-agent reinforcement learning (MARL) for discrete action spaces, yet its extension to continuous-action cooperative tasks remains challenging. Existing methods that approximate the counterfactual baseline via Monte Carlo sampling often introduce bias into policy gradients and fail to guarantee convergence to local optima, as the sampled actions may not have been sufficiently trained. To address these limitations, we propose SAFE, a novel MARL framework that employs a counterfactual baseline conditioned on a self-evolving default action sampled from each agent's experience buffer. This design naturally extends to continuous action spaces without relying on additional simulations, reward models, or environment-specific prior knowledge. The baseline accurately quantifies each agent's contribution, and introduces no bias into the deterministic policy gradient, ensuring convergence to local optima. Extensive experiments on cooperative vehicular tasks demonstrate that SAFE consistently outperforms state-of-the-art models.

Problem

• multi-agent reinforcement learning
• continuous action space
• counterfactual credit assignment
• policy gradient bias
• convergence

Innovation

• counterfactual credit assignment
• continuous action space
• self-evolving default action
• multi-agent reinforcement learning
• policy gradient convergence