Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearning

2026-07-30

AI Summary

This work addresses the inefficiency of existing machine unlearning methods that rely on sparse binary rewards, which yield weak learning signals and slow convergence, hindering effective selective knowledge removal. To overcome this limitation, the authors propose a reward decomposition framework that decouples verifiability from sparsity and introduce two novel dense reward functions: an exponential reward and a PageRank-inspired semantic weighting reward. By integrating concept counting with semantic importance analysis, the proposed approach substantially improves training efficiency on the RWKU benchmark—achieving up to a threefold speedup while maintaining comparable unlearning efficacy and model generalization. This study is the first to systematically demonstrate the critical role of reward design in the efficiency of machine unlearning.

Abstract

Machine unlearning seeks to selectively remove specific knowledge from trained language models without full retraining, a growing necessity under privacy regulations such as GDPR and the EU AI Act. Recent work has reformulated unlearning as a Reinforcement Learning with Verifiable Rewards (RLVR) problem, where models are optimized against verifiable rewards computed directly from their outputs. However, existing methods rely on sparse binary rewards that provide minimal learning signal, indicating only whether forbidden content was avoided, and limiting convergence speed. In this paper, we study how reward design affects unlearning efficiency within the Reinforcement Unlearning (RUL) framework. We introduce a principled reward decomposition framework that decouples verifiability from sparsity, and propose two new reward functions: an exponential reward that provides graded penalties based on the count of forbidden-concept occurrences, and a PageRank inspired reward that weights penalties by semantic importance. We conduct experiments on the Real World Knowledge Unlearning (RWKU) benchmark, demonstrating that both rewards consistently outperform the binary setting, while reaching similar forgetting performance up to $3\times$ faster and preserving general model utility. Our results show that reward design is a key driver of unlearning efficiency offering a practical path toward scalable and efficient machine unlearning.

Problem

• machine unlearning
• reinforcement learning
• reward design
• binary rewards
• language models

Innovation

• reward design
• reinforcement unlearning
• machine unlearning
• verifiable rewards
• non-binary rewards