Dual Path Attribution: Efficient Attribution for SwiGLU-Transformers through Layer-Wise Target Propagation

2026-03-20

AI Summary

Dense component attribution in large language models faces a fundamental trade-off between faithfulness and computational efficiency. This work proposes a dual-path attribution framework that achieves both high fidelity and efficiency within frozen SwiGLU-Transformer architectures. By propagating the target unembedding vector through a decomposed pathway in a single forward–backward pass, the method enables dense attribution with O(1) time complexity—eliminating the need for counterfactual samples and supporting long input sequences. Key technical innovations include inter-layer target propagation, structural linearization of the SwiGLU activation, and dual-path information flow tracing. Evaluated on standard interpretability benchmarks, the approach matches state-of-the-art faithfulness while substantially improving computational efficiency.

Technology Category

• Computer Vision: Interpretability, Explainability, and Transparency
• Natural Language Processing: (Large) Language Models
• Machine Learning: Mixture of Experts (MoE)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Understanding the internal mechanisms of transformer-based large language models (LLMs) is crucial for their reliable deployment and effective operation. While recent efforts have yielded a plethora of attribution methods attempting to balance faithfulness and computational efficiency, dense component attribution remains prohibitively expensive. In this work, we introduce Dual Path Attribution (DPA), a novel framework that faithfully traces information flow on the frozen transformer in one forward and one backward pass without requiring counterfactual examples. DPA analytically decomposes and linearizes the computational structure of the SwiGLU Transformers into distinct pathways along which it propagates a targeted unembedding vector to receive the effective representation at each residual position. This target-centric propagation achieves O(1) time complexity with respect to the number of model components, scaling to long input sequences and dense component attribution. Extensive experiments on standard interpretability benchmarks demonstrate that DPA achieves state-of-the-art faithfulness and unprecedented efficiency compared to existing baselines.

Problem

• attribution
• SwiGLU-Transformers
• computational efficiency
• faithfulness
• dense component attribution

Innovation

• Dual Path Attribution
• SwiGLU Transformers
• Layer-wise Target Propagation
• Efficient Attribution
• Faithful Interpretability