Provable Performance Bounds for Digital Twin-driven Deep Reinforcement Learning in Wireless Networks: A Novel Digital-Twin Bisimulation Metric

2025-02-25

AI Summary

Existing digital twin (DT)-driven deep reinforcement learning approaches lack provable guarantees on policy transfer performance, leading to unpredictable real-world deployment outcomes in wireless networks. Method: We propose the first Wasserstein-based DT bisimulation metric (DT-BSM), rigorously deriving a computable, provable upper bound on the real-world policy regret. To enable scalable deployment, we further design an empirical DT-BSM grounded in total variation distance and statistical sampling—balancing theoretical soundness with practical feasibility. Contribution/Results: We theoretically prove that the regret bound is jointly governed by the DT-BSM and the DT’s internal suboptimality. The empirical DT-BSM is shown to converge consistently, with explicit quantification of the relationship between sample size and estimation accuracy. Numerical experiments—first of their kind—demonstrate both the practical utility and tightness of the derived performance bound.

Technology Category

• Machine Learning: Online Learning & Bandits
• Search and Optimization: Sampling/Simulation-based Search
• Multiagent Systems: Other Foundations of Multi Agent Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Web Mining and Content Analysis: Web data generation and simulation
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Digital twin (DT)-driven deep reinforcement learning (DRL) has emerged as a promising paradigm for wireless network optimization, offering safe and efficient training environment for policy exploration. However, in theory existing methods cannot always guarantee real-world performance of DT-trained policies before actual deployment, due to the absence of a universal metric for assessing DT's ability to support reliable DRL training transferrable to physical networks. In this paper, we propose the DT bisimulation metric (DT-BSM), a novel metric based on the Wasserstein distance, to quantify the discrepancy between Markov decision processes (MDPs) in both the DT and the corresponding real-world wireless network environment. We prove that for any DT-trained policy, the sub-optimality of its performance (regret) in the real-world deployment is bounded by a weighted sum of the DT-BSM and its sub-optimality within the MDP in the DT. Then, a modified DT-BSM based on the total variation distance is also introduced to avoid the prohibitive calculation complexity of Wasserstein distance for large-scale wireless network scenarios. Further, to tackle the challenge of obtaining accurate transition probabilities of the MDP in real world for the DT-BSM calculation, we propose an empirical DT-BSM method based on statistical sampling. We prove that the empirical DT-BSM always converges to the desired theoretical one, and quantitatively establish the relationship between the required sample size and the target level of approximation accuracy. Numerical experiments validate this first theoretical finding on the provable and calculable performance bounds for DT-driven DRL.

Problem

• Quantify DT-driven DRL performance discrepancy
• Propose DT bisimulation metric for MDPs
• Ensure empirical DT-BSM convergence to theoretical

Innovation

• Digital Twin Bisimulation Metric
• Wasserstein Distance Application
• Empirical Statistical Sampling Method