Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs

2025-10-24

AI Summary

Existing continual knowledge editing methods for large language models suffer from error accumulation due to parameter interference, degrading both editing accuracy and generalization. This paper proposes a fine-grained neuron localization framework coupled with an entropy-guided dynamic sparse masking mechanism. First, neurons are functionally attributed and categorized into *knowledge-general* and *knowledge-specific* types. Then, only critical knowledge-specific neurons undergo sparse, adaptive weight updates—minimizing parameter perturbation while preserving model integrity. The method requires no full model retraining and maintains high editing success rates (+12.7%) and strong generalization stability (38.5% reduction in forgetting) over thousands of sequential edits—outperforming state-of-the-art approaches. Its core innovations lie in (i) interpretable, function-based neuron partitioning and (ii) an information-theoretic, sparsity-aware editing strategy that balances fidelity and plasticity.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Natural Language Processing: (Large) Language Models
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Lifelong knowledge editing enables continuous, precise updates to outdated knowledge in large language models (LLMs) without computationally expensive full retraining. However, existing methods often accumulate errors throughout the editing process, causing a gradual decline in both editing accuracy and generalization. To tackle this problem, we propose Neuron-Specific Masked Knowledge Editing (NMKE), a novel fine-grained editing framework that combines neuron-level attribution with dynamic sparse masking. Leveraging neuron functional attribution, we identify two key types of knowledge neurons, with knowledge-general neurons activating consistently across prompts and knowledge-specific neurons activating to specific prompts. NMKE further introduces an entropy-guided dynamic sparse mask, locating relevant neurons to the target knowledge. This strategy enables precise neuron-level knowledge editing with fewer parameter modifications. Experimental results from thousands of sequential edits demonstrate that NMKE outperforms existing methods in maintaining high editing success rates and preserving model general capabilities in lifelong editing.

Problem

• Enables continuous knowledge updates in LLMs without full retraining
• Addresses error accumulation during sequential editing processes
• Maintains editing accuracy while preserving model generalization capabilities

Innovation

• Dynamic sparse neuron masking for precise editing
• Identifies knowledge-general and knowledge-specific neurons
• Entropy-guided masking reduces parameter modification needs