Toward Ultra-Long-Horizon Sequential Model Editing

2026-01-30

AI Summary

This work addresses the critical issue of model collapse in existing model editing methods during long-sequence edits, which stems from the exponential growth of MLP weight norms and severely limits continual knowledge updating. For the first time, this study theoretically establishes the connection between editing-induced collapse and weight norm explosion. To mitigate this, the authors propose Norm-Anchor Scaling (NAS), a plug-and-play mechanism requiring only a single line of code and negligible computational overhead. NAS effectively curbs MLP weight growth through lightweight norm constraints. Integrated within the Locate-and-Edit framework, NAS significantly enhances editing stability across mainstream large language models, delaying the collapse point of representative algorithms by over fourfold and improving average editing performance by 72.2%, thereby demonstrating its efficiency and broad applicability.

Technology Category

• Natural Language Processing: Safety and Robustness
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Model editing has emerged as a practical approach for mitigating factual errors and outdated knowledge in large language models (LLMs). Among existing methods, the Locate-and-Edit (L&E) paradigm is the dominant framework: it locates MLP parameters implicated in expressing a target fact, and then performs a localized update to rewrite that fact. However, long sequences of edits often trigger abrupt model collapse in L&E beyond a critical point. We empirically identify a strong correlation between collapse and explosive growth of edited MLP weight norms, and formally prove that commonly used L&E update rules can induce exponential norm growth across sequential edits in the absence of explicit norm control. To address this issue, we propose Norm-Anchor Scaling NAS, a plug-and-play norm-constrained strategy. Across extensive experiments, NAS delays the collapse point of representative L&E algorithms by more than 4 times and yields a 72.2% average relative gain in editing performance, requiring only a single additional line of code and incurring negligible computational overhead.

Problem

• model editing
• Locate-and-Edit
• sequential edits
• model collapse
• weight norm explosion

Innovation

• model editing
• Locate-and-Edit
• norm explosion
• Norm-Anchor Scaling
• LLM updating