circuit tracing

This work addresses the challenge of ensuring safety and auditability in high-stakes applications of modern neural networks, which are often hindered by their opaque “black-box” nature. The authors propose a mechanistic interpretability framework that integrates Transformer circuit analysis, sparse autoencoders (SAEs), and neuro-symbolic reasoning to decompose complex activations into human-interpretable sparse features. This approach identifies critical computational units—such as induction heads—and establishes a mapping from neural representations to executable logical rules. By enabling causal interventions and targeted control over model behavior, the method achieves end-to-end symbolic explanations and controllability for the first time, significantly enhancing model transparency and intervenability without compromising performance.

Mechanistic interpretability faces foundational challenges—including methodological fragility, ambiguous scientific/engineering objectives, and salient socio-technical concerns. Method: This paper introduces the first three-dimensional problem taxonomy spanning concepts, methodologies, and ecosystem dynamics; proposes a goal-driven research paradigm prioritizing both scientific discovery and safety governance; and integrates computational neuroscience, formal verification, causal reasoning, and human-in-the-loop analysis to advance tooling from phenomenological description toward mechanistic modeling. Contribution/Results: We distill over a dozen high-priority open problems, establish a community-aligned research agenda, and thereby significantly accelerate foundational theory development for trustworthy AI and practical mechanistic analysis of large language models.

Existing interpretability methods for vision models predominantly focus on neuron activations, lacking a mechanistic understanding of how information propagates through the model. This work proposes Visual Circuit Discovery (Vi-CD), the first method to introduce edge-level mechanistic circuit analysis into Vision Transformers. By constructing an edge-based computational graph, Vi-CD automatically identifies class-specific information pathways relevant to particular tasks. The approach not only uncovers the internal information routing mechanisms of vision models but also successfully locates adversarial circuits in CLIP that underlie typographic attacks. Furthermore, targeted interventions on these discovered circuits effectively mitigate harmful behaviors, enabling transparent and controllable manipulation of visual model decision processes.

This work addresses the challenge of interpreting neural network training dynamics. We propose “circuit-tuning”: first modeling node redundancy via intrinsic dimensionality to identify task-relevant computational circuits and mask redundant connections; then updating only critical parameters to reveal the self-organizing structural evolution during learning. This establishes the first interpretable, fine-grained framework for analyzing training dynamics, enabling task-directed structured sparse fine-tuning. Experiments demonstrate superior multi-task performance over standard fine-tuning, with a 72% average reduction in updated parameters while preserving generalization. The method further provides dynamic visual evidence of circuit evolution throughout training. Our core innovation unifies redundancy quantification and circuit-driven iterative optimization within an interpretable training analysis paradigm.

The causal origins of interpretable units—such as induction heads—in large language models remain poorly understood. This work proposes a scalable mechanistic data attribution framework that integrates influence functions with causal interventions to establish, for the first time, direct causal links between specific training examples and the emergence of such interpretable components. The study reveals that structured repetitive data plays a catalytic role in circuit formation and demonstrates a direct functional relationship between induction heads and in-context learning capabilities. By selectively intervening on a small set of high-influence training samples, the emergence of attention heads can be significantly modulated. Furthermore, the proposed data augmentation strategy consistently accelerates circuit convergence across different model scales.

This work addresses the limited generalizability of traditional mechanistic interpretability methods, which often rely on task-specific customization. It proposes a novel, general-purpose circuit discovery framework that integrates reinforcement learning with causal intervention. Specifically, a PPO agent performs zero-ablation operations across all 144 attention heads of GPT-2 Small and automatically identifies critical heads using a contrastive reward signal—eliminating the need for task-specific analysis pipelines and enabling cross-task transfer. Experimental results demonstrate that the method achieves near-oracle performance both on training tasks and unseen tasks: in the absence of task-specific signals, the best five runs recover 96% of the oracle upper bound and accurately identify both key and redundant attention heads previously documented in the literature.

This study investigates the internal mechanisms of jet classifiers in particle physics, focusing on the interpretability of the Particle Transformer in top-quark jet tagging. For the first time, interpretability methods from natural language processing—namely zero ablation, path patching, manifold perturbation, and residual stream linear probing—are adapted to high-energy physics, with comparisons drawn between energy correlation functions and N-subjettiness bases. The analysis reveals that a sparse circuit comprising only six attention heads suffices to reproduce the model’s primary performance. Residual streams exhibit a strong preference for energy correlation bases, particularly in encoding two-prong substructure. Classification decisions emerge from an effective basis transformation following early signal saturation. These findings demonstrate the model’s capacity to autonomously learn physically meaningful representations under unsupervised pretraining.

This work addresses the challenge in mechanistic interpretability that, despite progress in circuit localization, component-level functional explanations remain manual and lack standardization. To this end, we propose HyVE, a novel framework that introduces language model agents into circuit explanation tasks. HyVE iteratively performs observation, hypothesis generation, and causal verification to automatically produce both component-level interpretations and circuit-level task descriptions. We construct AgenticInterpBench, the first benchmark tailored for agent-based interpretability, and evaluate HyVE across four mainstream language model architectures, demonstrating its ability to generate high-quality explanations. Our experiments reveal that causal verification constitutes the primary performance bottleneck, and we further showcase HyVE’s practical utility through a case study on arithmetic circuits in Llama-3-8B.

This work addresses the lack of a verifiable, comparable, and composable mechanistic framework for evaluating neural model interpretability. It proposes the first formal framework grounded in compositionality and the Minimum Description Length principle, leveraging category theory to decompose models into syntax–semantics mapping pairs. Fidelity of explanations is ensured through commutative diagram consistency constraints. Within this framework, interpretability is formulated as an optimization problem balancing fidelity against complexity, and a compression-based distillation algorithm is introduced to systematically simplify model structure without altering its functional behavior. The approach unifies existing mechanistic explanation methods as special cases and demonstrates that, under a parsimony criterion, syntactic compression yields more concise and cognitively plausible explanations, thereby establishing a theoretical foundation for the quantitative evaluation and automated discovery of interpretability.

Current mechanistic interpretability lacks a precise definition of “valid explanation” and relies heavily on manual construction, hindering scalability. This work formalizes, for the first time, the problem of *explanatory equivalence*—determining whether distinct neural networks implement the same underlying algorithmic mechanism without requiring an explicit description of that mechanism. By establishing theoretical connections among explanations, circuits, and internal representations, the study proposes a method for assessing equivalence based on representational similarity analysis and validates its efficacy on Transformer models. Furthermore, it provides necessary and sufficient conditions for explanatory equivalence, thereby offering a rigorous evaluation framework and a pathway toward automated discovery in mechanistic interpretability.