Policy Optimality Measurement for Multi-Vehicle Decision-Making: From Extrinsic Indicators to Intrinsic Quality

2026-08-02

AI Summary

This work addresses the limitations of existing extrinsic-metric-based evaluation methods in multi-agent reinforcement learning (MARL), which often fail to uncover intrinsic policy degradation and algorithmic blind spots. We propose the first model-free, intrinsic framework for assessing policy quality by leveraging fully converged Monte Carlo tree search as an asymptotically optimal benchmark. Building upon this, we define a bounded policy optimality score via forward KL divergence and semantically decouple it into horizontal and vertical dimensions, enabling fine-grained and interpretable policy diagnosis. This approach transforms black-box evaluation into a “semantic microscope,” effectively revealing directional biases and temporal averaging pitfalls prevalent in mainstream MARL architectures. Consequently, hyperparameter tuning shifts from heuristic trial-and-error toward visualized trajectory optimization, facilitating precise policy analysis across both spatial and temporal dimensions.

Abstract

Evaluating Multi-Agent Reinforcement Learning (MARL) policies in autonomous driving fundamentally relies on extrinsic statistical indicators (e.g., reward curves and success rates), which often mask intrinsic policy degradation and algorithmic blind spots. To break this black-box evaluation, this letter proposes a novel information-theoretic diagnostic framework. By leveraging a fully converged Monte Carlo Tree Search (MCTS) as an asymptotic oracle, we establish a theoretical ground-truth baseline distribution. We formulate a bounded policy optimality score ($\mathcal{M}_{opt}$) using the forward KL divergence to rigorously penalize fatal collaborative omissions. Crucially, we semantically decouple this metric into lateral and longitudinal dimensions, creating a granular "semantic microscope". Extensive spatial and temporal diagnostics on state-of-the-art MARL architectures and exploration mechanisms demonstrate that our framework conclusively exposes hidden directional biases, identifies temporal average-policy traps, and transforms heuristic hyperparameter tuning into a visually trackable trajectory optimization. This framework establishes a rigorous, model-agnostic standard for benchmarking intrinsic multi-agent policy quality.

Problem

• Multi-Agent Reinforcement Learning
• Policy Evaluation
• Autonomous Driving
• Intrinsic Quality
• Optimality Measurement

Innovation

• policy optimality
• information-theoretic evaluation
• Monte Carlo Tree Search
• multi-agent reinforcement learning
• semantic decoupling