Robust Asynchronous Q-Learning under Reward and State Corruption via Batching

2026-07-22

AI Summary

This work addresses the degradation of asynchronous Q-learning under adversarial perturbations that simultaneously corrupt both state observations and reward signals. To tackle this challenge, the authors propose BR-Async-Q, a novel algorithm that partitions online data into batches and constructs a robust Bellman optimality operator based on the Huber contamination model. This approach provides the first theoretical robustness guarantee for asynchronous Q-learning in settings where states and rewards are jointly contaminated. In the special case of reward-only corruption, BR-Async-Q achieves minimax-optimal dependence on the contamination ratio. Its ℓ∞ error bound matches that of standard Q-learning up to an additive term proportional to the contamination level, thereby substantially enhancing learning stability in adversarial environments.

Abstract

Motivated by reinforcement learning in harsh environments, we consider the problem of learning an optimal policy subject to adversarially corrupted feedback. Specifically, at each time-step, an adversary can perturb both the reward and state observations of the learner following the Huber contamination model. To defend against such data corruption, we propose {\texttt{BR-Async-Q}}: a novel, epoch-based, robust \(Q\)-learning algorithm built upon two key ideas: (i) partitioning the online data stream into batches to reduce variance, and (ii) constructing robust estimates of the Bellman optimality operator using such batched data. We prove a high-probability $\ell_\infty$ error bound for {\texttt{BR-Async-Q}} that matches that for vanilla \(Q\)-learning, up to a small additive term that scales with the fraction of corrupted samples. To our knowledge, this provides the first robustness guarantee for asynchronous \(Q\)-learning subject to both reward and state corruption. Furthermore, when only rewards are corrupted, the dependence of our algorithm's bound on the corruption fraction is minimax optimal.

Problem

• robust reinforcement learning
• adversarial corruption
• Q-learning
• state corruption
• reward corruption

Innovation

• robust reinforcement learning
• asynchronous Q-learning
• adversarial corruption
• batching
• Bellman operator