ManiEdit: Sequential Unstructured Knowledge Editing for Language Models from a Manifold Perspective

2026-09-27

AI Summary

This study addresses the issues of catastrophic forgetting and capability degradation in large language models during the sequential editing of unstructured knowledge. To this end, we propose a manifold-aware autoregressive framework that formulates knowledge editing as local displacement on a submanifold. By leveraging pivot point localization, energy-weighted penalization, and recursive null-space alignment, the method precisely updates target knowledge while preserving the underlying manifold structure within the parameter space. Experimental results demonstrate that our approach significantly outperforms existing methods across multiple benchmarks, enabling efficient and continuous knowledge updates while retaining near-original general capabilities.

Abstract

Large language models (LLMs) inevitably generate some incorrect or outdated content, necessitating efficient and precise mechanisms for continual knowledge updates. However, existing model editing methods struggle to sequentially edit unstructured long-form knowledge, suffering from severe edit forgetting and degradation of general capabilities. To address these challenges, we reframe knowledge editing from a manifold perspective, viewing it as a localized displacement of an edit sub-manifold within the global knowledge manifold. Under this formulation, the problem can be decomposed into two key questions: (i) how to identify representative edit points that effectively anchor the edit sub-manifold, and (ii) how to preserve the remaining manifold structure during the sub-manifold displacement process. Based on this perspective, we propose ManiEdit, a novel manifold-aware autoregressive editing framework consisting of two core components. Pivot Localization addresses the mediocre-point dilemma by identifying high-leverage pivots to anchor the edit sub-manifold. Manifold-Aware Preservation preserves different knowledge types through an energy-weighted penalty combined with recursive null-space alignment. Experiments on two base LLMs and four unstructured editing benchmarks demonstrate that ManiEdit achieves state-of-the-art performance, outperforming the strongest baseline by up to +27.81 BERTScore and +8.50 ROUGE-L, while maintaining near-original general capabilities across six representative downstream tasks. Our code is available at: https://github.com/Areyliu/ManiEdit

Problem

• Knowledge Editing
• Large Language Models
• Sequential Editing
• Unstructured Knowledge
• Edit Forgetting

Innovation

• Manifold Perspective
• Knowledge Editing
• Pivot Localization
• Null-space Alignment
• Unstructured Knowledge