Knowledge Editing with Subspace-Aware Key-Value Mappings

2025-09-29

AI Summary

To address the challenges of high edit interference and weak knowledge retention in correcting factual errors of large language models (LLMs), this paper proposes a subspace-aware knowledge editing method. Built upon the locate-then-edit paradigm, our approach employs subspace analysis to identify critical feature directions associated with the target fact and applies constraints exclusively within the key-value mappings of MLP layers—thereby enabling precise, localized modifications. The core innovation lies in a learnable subspace projection mechanism that strictly confines editing operations to semantically sensitive dimensions, substantially reducing perturbation to unrelated knowledge. Experiments on LLaMA-3-8B, GPT-J-6B, and Qwen2.5-7B demonstrate that our method achieves over 95% edit success rate while improving knowledge retention by an average of 23.6%, significantly outperforming existing baselines.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Fact-Checking / Misinformation Detection (NLP Focus)Computer Vision: Large Vision Models

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Knowledge editing aims to efficiently correct factual errors in Language Models (LMs). The popular locate-then-edit approach modifies an MLP layer by finding an optimal mapping between its input vector (key) and output vector (value) that leads to the expression of the edited knowledge. However, existing methods without any constraints on the key and value vectors cause significant perturbations to the edited model. To address this, we propose Subspace Knowledge Edit (SUIT), a method that identifies and modifies only the subspace of critical features relevant to the edit. Our empirical results on LLaMA-3-8B, GPT-J-6B, and Qwen2.5-7B models show that SUIT dramatically improves knowledge preservation over strong baselines while maintaining high edit efficacy. This effectiveness confirms that SUIT successfully identifies the critical subspace for the edit. Further analyses provide additional validation for our approach. The source code and data will be released to the public upon publication of the paper.

Problem

• Correct factual errors in language models efficiently
• Minimize perturbations during knowledge editing process
• Identify critical feature subspace for targeted edits

Innovation

• Modifies only critical feature subspace for edits
• Uses subspace-aware key-value mappings in MLP
• Improves knowledge preservation while maintaining efficacy