Sparse Weight Decomposition for Efficient Circuit Extraction

2026-08-04

AI Summary

This work addresses the lack of interpretable circuit units in dense pretrained Transformers and the computational overhead and representational distortion introduced by existing sparsification methods that require additional training. The authors propose Sparse Weight Decomposition (SWD), which reparameterizes linear projection weights into two sparse factors sharing an intermediate coordinate space that serves as addressable, interpretable circuit units. SWD enables direct, standard circuit extraction without training surrogate models or using extra data. Evaluated on GPT-2, Qwen2.5, and Qwen3.5-27B, SWD achieves fidelity comparable to Transcoder using less than 1% of the original training data, while requiring fewer activated edges and units to realize the same circuit objectives. Full-model replacement followed by fine-tuning preserves performance, marking the first zero-data, high-fidelity, end-to-end circuit extraction method and significantly advancing mechanistic interpretability analysis.

Abstract

Dense pretrained transformers do not naturally expose interpretable units for circuit extraction. Existing approaches obtain such units by learning auxiliary sparse representations or training sparse models, incurring substantial additional computation while potentially introducing a fidelity gap between the representation being analyzed and the original pretrained model. We propose Sparse Weight Decomposition (SWD), which reparameterizes pretrained linear projections by factorizing each weight matrix into two sparse factors whose shared intermediate coordinates serve as individually addressable circuit units. Without training a separate replacement network, this parametric representation supports the same scoring, selection, and ablation circuit extraction workflow used for methods that learn sparse features. Across single-matrix replacements, SWD matches the held-out fidelity achieved by Transcoder and other strong baselines while using less than 1% of the data that those baselines use to train their replacements. For matched replacement fidelity, SWD reaches the same circuit sufficiency and necessity targets with fewer active read/write edges and selected units across tasks on GPT-2, Qwen2.5, and Qwen3.5-27B. We further show that SWD remains effective for full-model replacement of all attention and MLP weight matrices after fine-tuning the nonzero factor values. Finally, SWD also features a zero-data variant, allowing broader use of mechanistic interpretability analysis (e.g., per-step analysis).

Problem

• circuit extraction
• interpretable units
• sparse representation
• pretrained transformers
• fidelity gap

Innovation

• Sparse Weight Decomposition
• circuit extraction
• mechanistic interpretability
• weight factorization
• zero-data analysis