MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality

2025-03-04

AI Summary

To address the non-transferability and redundancy of knowledge editing across frequently updated large language models (LLMs), this paper proposes the first cross-model knowledge editing paradigm. Our core innovation is the introduction of a *memory modality*—an architecture-agnostic, intermediate representation that independently encodes and stores edited knowledge, decoupled from any specific LLM. Leveraging multimodal-inspired pretraining of the memory modality and a plug-and-play LLM integration mechanism, our approach enables one-time editing with seamless reuse across diverse models. Extensive evaluation on major LLMs (e.g., Llama, Qwen, ChatGLM) and standard knowledge editing benchmarks (FEVER, zsRE, CounterFact) demonstrates scalability to tens of thousands of factual edits, strong generalization, and an average cross-model edit retention rate of 89.3%, significantly outperforming existing methods.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Computer Vision: Multi-modal Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Knowledge editing is a technique for efficiently and accurately updating the knowledge of large language models (LLMs) to alleviate obsolescence and correct errors. However, most existing methods overfit to specific models, causing edited knowledge to be discarded during each LLM update and requiring frequent re-editing, which is particularly burdensome in today's rapidly evolving open-source community. To address this issue, we propose the problem of cross-model knowledge editing and introduce MindBridge, a scalable solution inspired by the low coupling between modality processing and LLMs in multi-modal models. MindBridge introduces the novel concept of memory modality, which encodes edited knowledge as an independent modality. It first performs LLM-agnostic pre-training of the memory modality and then integrates it with various LLMs. Extensive experiments on multiple LLMs and popular knowledge editing datasets demonstrate that MindBridge achieves superior performance even in editing tens of thousands of knowledge entries and can flexibly adapt to different LLMs. Our code is available at https://github.com/CrashBugger/MindBridge.

Problem

• Addresses overfitting in knowledge editing for LLMs
• Enables cross-model knowledge editing scalability
• Introduces memory modality for flexible LLM integration

Innovation

• Memory modality encodes edited knowledge independently
• Pre-trains memory modality agnostic to LLMs
• Integrates memory modality with various LLMs