ForgetBench: Benchmarking Forgetting Dynamics of Long-Term Parametric Memory in Language Models

2026-07-29

AI Summary

This work addresses the challenge of catastrophic forgetting in large language models during continual knowledge updates, a phenomenon inadequately captured by existing evaluation protocols due to their inability to model the temporal dynamics of memory loss. To this end, we introduce ForgetBench, the first benchmark specifically designed for temporally aware evaluation of continuous knowledge editing. ForgetBench employs two complementary paradigms—concept-based and scenario-based questioning—to disentangle the retention of isolated factual knowledge from structured relational knowledge. Built upon a unified framework for analyzing long-term memory evolution, it integrates serialized knowledge streams, multi-stage editing simulations, and temporal decay modeling to quantitatively assess both memory retention strength and cross-instance stability. Empirical results reveal that current editing methods struggle to simultaneously preserve long-term memory and maintain generalization capability, underscoring the urgent need for more robust temporal memory mechanisms in large language models.

Abstract

Large language models (LLMs) have demonstrated strong capabilities in knowledge acquisition and reasoning, yet their ability to retain previously acquired knowledge under repeated updates remains insufficiently understood. Existing evaluation paradigms primarily focus on single-step reasoning or static knowledge editing, which fail to capture the temporal dynamics of knowledge retention and degradation during continual model modification. In this work, we propose ForgetBench, a benchmark designed to systematically characterize forgetting behavior in LLMs under continual knowledge editing. ForgetBench introduces two complementary evaluation paradigms, namely concept-based QA and scenario-based QA, to disentangle isolated factual retention from structured relational knowledge preservation. Building upon a sequential editing framework, we construct temporally ordered knowledge streams and evaluate model behavior across multiple editing stages. To quantitatively analyze long-term retention dynamics, we further introduce a unified evaluation framework that models knowledge evolution over time, enabling the measurement of temporal decay, retention strength, and cross-instance stability. Extensive experiments across diverse models and editing methods demonstrate that existing approaches fail to strike a balance between long-term retention and generalization quality. Our findings highlight the need for more robust memory mechanisms that can effectively acquire, update, and preserve knowledge over time in future LLMs. Code will be released upon acceptance.

Problem

• forgetting dynamics
• long-term memory
• continual knowledge editing
• knowledge retention
• language models

Innovation

• forgetting dynamics
• continual knowledge editing
• long-term memory
• temporal evaluation
• language models