Randomized Least Squares Value Iteration itself is Joint Differentially Private

2026-06-01

AI Summary

This work addresses the challenge of preserving user privacy in sensitive applications of reinforcement learning—such as healthcare and recommendation systems—without introducing additional noise. Focusing on tabular Markov Decision Processes (MDPs), the study investigates the intrinsic privacy properties of Randomized Least-Squares Value Iteration (RLSVI) and establishes, for the first time, that its inherent exploration noise alone satisfies $(\varepsilon(\delta), \delta)$-joint differential privacy, eliminating the need for explicit perturbations. By leveraging joint differential privacy analysis, probabilistic bounding techniques, and privacy amplification, the paper reveals a fundamental connection between exploration mechanisms and privacy guarantees. Under standard parameters—$S$ states, $A$ actions, horizon $H$, and $K$ episodes—the algorithm achieves a privacy bound of $\varepsilon(\delta) = \frac{2AK}{H^2 \log(2HSA)} + 2\sqrt{\frac{2AK \log(1/\delta)}{H^2 \log(2HSA)}}$.

Abstract

As reinforcement learning (RL) increasingly applies to sensitive domains, such as health care and recommendation systems, privacy-preserving techniques have become essential to protect users' sensitive information. We investigate privacy-preserving RL under an episodic setting, focusing on algorithms based on randomized exploration, such as Randomized Least Squares Value Iteration (RLSVI). The overall goal is to study how randomized exploration interacts with the injected noise required by privacy mechanisms. In this work, we show a new privacy analysis that characterizes how the noise in RLSVI set for exploration simultaneously provides privacy protection. Specifically, we prove that RLSVI is $(\varepsilon(δ),δ)$-joint differentially private in tabular MDP as is with $\varepsilon(δ) = \frac{2AK}{H^2\log(2HSA)} + 2\sqrt{\frac{2AK\log(1/δ)}{H^2\log(2HSA)}}$, where $S$ and $A$ are the number of states and actions respectively, $H$ is the length of an episode and $K$ is the number of episodes.

Problem

• privacy-preserving reinforcement learning
• randomized exploration
• joint differential privacy
• episodic MDP
• sensitive information protection

Innovation

• Randomized Least Squares Value Iteration
• Joint Differential Privacy
• Privacy-Preserving Reinforcement Learning
• Exploration-Privacy Trade-off
• Episodic MDP