Model Merging for Knowledge Editing

2025-06-14

AI Summary

To address performance degradation and diminished general capabilities in large language models (LLMs) during sequential knowledge editing, this paper proposes a two-stage knowledge updating framework. First, robust supervised fine-tuning (R-SFT) internalizes new knowledge; second, the fine-tuned model is parameter-space fused with the original base model. This work introduces the novel “fine-tuning + fusion” paradigm—requiring no architectural modifications—enabling high-accuracy sequential editing while preserving pre-edit capabilities. Evaluated under a rigorous knowledge editing benchmark across multiple rounds of sequential edits, our method achieves a 23.5% improvement in knowledge correction accuracy and constrains performance decay on original tasks to within 0.8%, substantially outperforming state-of-the-art approaches.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Safety and RobustnessSearch and Optimization: Learning to Search

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves. While existing knowledge editing approaches offer various solutions for knowledge updating, they often struggle with sequential editing scenarios and harm the general capabilities of the model, thereby significantly hampering their practical applicability. This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing. Our method first fine-tunes the LLM to internalize new knowledge fully, then merges the fine-tuned model with the original foundation model to preserve newly acquired knowledge and general capabilities. Experimental results demonstrate that our approach significantly outperforms existing methods in sequential editing while better preserving the original performance of the model, all without requiring any architectural changes. Code is available at: https://github.com/Applied-Machine-Learning-Lab/MM4KE.

Problem

• Updating LLMs to maintain accurate knowledge over time
• Preserving model capabilities during sequential knowledge editing
• Avoiding performance degradation in existing knowledge editing methods

Innovation

• Two-stage framework for knowledge editing
• Combines robust supervised fine-tuning with model merging
• Preserves new knowledge and general capabilities