Understanding Language Model Circuits through Knowledge Editing

2024-06-25

AI Summary

This study investigates the knowledge organization mechanisms underlying critical behavioral circuits in GPT-2. Addressing the challenges of locality and editability in language model knowledge representation, we propose a causal mediation–based knowledge editing framework that integrates ROME-based circuit localization, multi-layer probing, and attribution analysis to systematically evaluate editing response patterns, knowledge distribution breadth, and structural robustness. Our key contributions are: (1) the first empirical demonstration that factual knowledge is non-localized—distributed across layers and heads via collaborative semantic encoding; (2) evidence that single-point edits induce consistent semantic updates throughout the entire transformer stack; and (3) the finding that circuits exhibit both functional specificity and knowledge redundancy. Consequently, we formally define a circuit’s “meaning” as an intervenable, generalizable semantic functional unit—establishing a novel paradigm for model interpretability and safe, controllable knowledge editing.

Technology Category

• Knowledge Representation and Reasoning: Action, Change, and Causality
• Machine Learning: Probabilistic Circuits and Graphical Models
• Natural Language Processing: Safety and Robustness

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding

Abstract

Recent advances in language model interpretability have identified circuits, critical subnetworks that replicate model behaviors, yet how knowledge is structured within these crucial subnetworks remains opaque. To gain an understanding toward the knowledge in the circuits, we conduct systematic knowledge editing experiments on the circuits of the GPT-2 language model. Our analysis reveals intriguing patterns in how circuits respond to editing attempts, the extent of knowledge distribution across network components, and the architectural composition of knowledge-bearing circuits. These findings offer insights into the complex relationship between model circuits and knowledge representation, deepening the understanding of how information is organized within language models. Our findings offer novel insights into the ``meanings'' of the circuits, and introduce directions for further interpretability and safety research of language models.

Problem

• How knowledge is structured within critical model subnetworks
• How circuits respond to systematic knowledge editing attempts
• The architectural composition of knowledge-bearing circuits in models

Innovation

• Systematic knowledge editing on GPT-2 circuits
• Analyzing circuit responses to editing attempts
• Exploring knowledge distribution in network components