TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning

2026-07-23

AI Summary

This work addresses the challenge in offline reinforcement learning that existing metrics inadequately evaluate trajectory-level data removal, often conflating forgetting, residual memorization, and policy degradation. To this end, the paper introduces TOUR, a comprehensive benchmark that integrates trajectory partitioning, non-member controls, retraining references, performance anchors, and diverse privacy attacks—including action error, representation similarity, and query-limited attacks—to systematically assess forgetting quality. Empirical results demonstrate that single membership scores are unreliable; the efficacy of deletion methods is highly environment-dependent (e.g., D4RL versus AntMaze); retraining and fine-tuning generally outperform GA+Refit; and robust evaluation necessitates a holistic consideration of attack types, utility preservation, and architectural evidence.

Abstract

Offline Reinforcement Learning (RL) agents are trained on fixed behavioral trajectories, which makes trajectory-level deletion important when selected data must be removed after training. Evaluating such deletion is difficult because a lower membership score can reflect trajectory removal, residual memorization visible to another attack, or policy collapse that destroys useful behavior. We introduce Trajectory-level memOrization and Unlearning in offline RL (TOUR), a benchmark that combines trajectory-level partitioning, matched non-member controls, retraining references, retained-performance anchors, and multi-attack privacy auditing. Across D4RL locomotion experiments and an exploratory AntMaze extension, TOUR shows that common deletion baselines have environment-dependent privacy-utility behavior. Retraining and fine-tuning often provide stronger retained-utility references than uniform GA+Refit, while TrajDeleter remains a useful comparator but is not uniformly stronger under the same audit. Reference-model, threshold, deviation, equivalence, action-error, representation-based, and query-limited attacks further show that a single likelihood-based membership score can overstate deletion quality. In the evaluated settings, conclusions about offline RL unlearning are therefore not stable under single-score auditing. They depend on matched non-member construction, retraining-relative calibration, attack family, retained utility, and explicit scope for diagnostic architecture or component-level evidence.

Problem

• offline reinforcement learning
• machine unlearning
• trajectory-level deletion
• membership inference
• privacy auditing

Innovation

• trajectory-level unlearning
• offline reinforcement learning
• membership inference
• privacy auditing
• benchmark