Diffusion Model-based Reinforcement Learning for Version Age of Information Scheduling: Average and Tail-Risk-Sensitive Control

2026-01-26

AI Summary

This work addresses the limitation of existing approaches in multi-user wireless status update systems, which focus solely on minimizing the average semantic Age of Information (VAoI) while neglecting tail risks induced by stochastic packet arrivals and unreliable channels. To this end, we propose the first framework integrating diffusion models with distributional reinforcement learning to jointly optimize both the average VAoI and tail risk under a long-term transmission cost constraint. The proposed RS-D3SAC algorithm combines a diffusion-based policy network with a quantile distributional critic, explicitly modeling the return distribution of VAoI and optimizing tail risk via Conditional Value-at-Risk (CVaR). Its base variant, D2SAC, also effectively reduces average VAoI. Experimental results demonstrate that RS-D3SAC significantly lowers CVaR without compromising mean performance, highlighting the critical role of the distributional critic in tail risk control and the stability advantage of diffusion policies.

Technology Category

• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• Web Mining and Content Analysis: Content-based information diffusion
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Ensuring timely and semantically accurate information delivery is critical in real-time wireless systems. While Age of Information (AoI) quantifies temporal freshness, Version Age of Information (VAoI) captures semantic staleness by accounting for version evolution between transmitters and receivers. Existing VAoI scheduling approaches primarily focus on minimizing average VAoI, overlooking rare but severe staleness events that can compromise reliability under stochastic packet arrivals and unreliable channels. This paper investigates both average-oriented and tail-risk-sensitive VAoI scheduling in a multi-user status update system with long-term transmission cost constraints. We first formulate the average VAoI minimization problem as a constrained Markov decision process and introduce a deep diffusion-based Soft Actor-Critic (D2SAC) algorithm. By generating actions through a diffusion-based denoising process, D2SAC enhances policy expressiveness and establishes a strong baseline for mean performance. Building on this foundation, we put forth RS-D3SAC, a risk-sensitive deep distributional diffusion-based Soft Actor-Critic algorithm. RS-D3SAC integrates a diffusion-based actor with a quantile-based distributional critic, explicitly modeling the full VAoI return distribution. This enables principled tail-risk optimization via Conditional Value-at-Risk (CVaR) while satisfying long-term transmission cost constraints. Extensive simulations show that, while D2SAC reduces average VAoI, RS-D3SAC consistently achieves substantial reductions in CVaR without sacrificing mean performance. The dominant gain in tail-risk reduction stems from the distributional critic, with the diffusion-based actor providing complementary refinement to stabilize and enrich policy decisions, highlighting their effectiveness for robust and risk-aware VAoI scheduling in multi-user wireless systems.

Problem

• Version Age of Information
• tail-risk-sensitive control
• semantic staleness
• multi-user scheduling
• wireless status update

Innovation

• Diffusion-based Reinforcement Learning
• Version Age of Information
• Tail-risk-sensitive Control
• Distributional Reinforcement Learning
• Conditional Value-at-Risk