Concept-Based Mechanistic Interpretability Using Structured Knowledge Graphs

2025-07-08

AI Summary

Conventional concept-based interpretability methods provide only local explanations, failing to capture global neural circuit dynamics in deep models. Method: We propose the first concept-driven framework for global neural circuit analysis, introducing a model-agnostic, hierarchical concept analysis pipeline that quantifies the emergence, interaction, and propagation of semantic concepts across layers; we further design BAGEL—a visualization platform that encodes concept–class relationships as structured knowledge graphs to diagnose spurious correlations and data biases. Contributions/Results: (1) First systematic, end-to-end tracking of high-level semantic concepts throughout deep model internals; (2) Discovery of decision-critical latent neural circuits and information flow patterns; (3) Effective identification of generalization failures induced by data bias, substantially improving explanation fidelity and debugging capability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning
• Computer Vision: Visual Reasoning & Symbolic Representations
• Machine Learning: Probabilistic Circuits and Graphical Models

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

While concept-based interpretability methods have traditionally focused on local explanations of neural network predictions, we propose a novel framework and interactive tool that extends these methods into the domain of mechanistic interpretability. Our approach enables a global dissection of model behavior by analyzing how high-level semantic attributes (referred to as concepts) emerge, interact, and propagate through internal model components. Unlike prior work that isolates individual neurons or predictions, our framework systematically quantifies how semantic concepts are represented across layers, revealing latent circuits and information flow that underlie model decision-making. A key innovation is our visualization platform that we named BAGEL (for Bias Analysis with a Graph for global Explanation Layers), which presents these insights in a structured knowledge graph, allowing users to explore concept-class relationships, identify spurious correlations, and enhance model trustworthiness. Our framework is model-agnostic, scalable, and contributes to a deeper understanding of how deep learning models generalize (or fail to) in the presence of dataset biases. The demonstration is available at https://knowledge-graph-ui-4a7cb5.gitlab.io/.

Problem

• Extends concept-based interpretability to mechanistic analysis of neural networks
• Analyzes global concept interactions and propagation in model components
• Identifies latent circuits and biases affecting model decision-making

Innovation

• Global model behavior analysis via semantic concepts
• Structured knowledge graph visualization tool BAGEL
• Model-agnostic framework for bias and generalization analysis