Leak-Resistant Unlearning: A New Benchmark for Evaluating Multi-Hop Reasoning Consistency and Recovery Robustness

2026-08-05

AI Summary

Existing unlearning methods for large language models often fail to prevent leakage of sensitive knowledge in multi-hop reasoning scenarios and exhibit limited robustness against lightweight recovery attacks, with no systematic evaluation framework for unlearning robustness. This work introduces the first unified benchmark that incorporates multi-hop reasoning consistency and resilience to recovery attacks, evaluating six unlearning methods across three model architectures using two custom datasets. Through comprehensive assessments involving multi-hop question answering and post-adaptation recovery attacks, the study reveals a fundamental trade-off among unlearning efficacy, resistance to knowledge recovery, and retained model utility. The findings highlight the fragility of current approaches and establish a reliable foundation for future research on robust unlearning.

Abstract

Benchmarking machine unlearning methods is critical to understand whether sensitive knowledge is removed from large language models (LLMs) or not. Current unlearning benchmarks include mainly single-hop questions and a narrow set of multi-hop questions. Although effective, they still face two challenges. (1) Knowledge is not isolated, whereby diverse multi-hop reasoning paths can potentially induce knowledge leakage than normal queries. (2) Unlearning may be fragile: unlearned knowledge can be partially recovered through recovery attacks such as lightweight post-unlearning adaptation, making static evaluation insufficient. Therefore, in this paper, we introduce \unlearning as a novel benchmark to understand robust LLM knowledge removal across diverse reasoning paths and recovery attacks. We experiment with this benchmark on 3 models, 6 unlearning methods, and 2 carefully curated datasets. Results show that existing methods are vulnerable to multi-hop reasoning paths and recovery attacks. We further explore the trade-off among forget quality, robustness, and model utility for LLM unlearning.

Problem

• machine unlearning
• multi-hop reasoning
• knowledge leakage
• recovery attacks
• LLM robustness

Innovation

• machine unlearning
• multi-hop reasoning
• recovery robustness
• knowledge leakage
• LLM benchmarking