Findings of the BlackboxNLP 2025 Shared Task: Localizing Circuits and Causal Variables in Language Models

2025-11-23

AI Summary

This study addresses the lack of standardized evaluation protocols for two core tasks in mechanistic interpretability (MI) of language models: circuit localization (identifying causal components and their interactions) and causal variable localization (mapping neural activations to human-interpretable features). To bridge this gap, we extend the Mechanistic Interpretability Benchmark (MIB) into the first open, unified evaluation platform supporting both tracks under consistent metrics, enabling fair, cross-team benchmarking. Methodologically, our framework integrates ensemble learning, structured regularization, low-dimensional projection, and nonlinear feature mapping to yield robust, interpretable representations of hidden-layer activations. Empirical results demonstrate significant improvements over existing baselines in both circuit localization accuracy and causal variable interpretability. The MIB Leaderboard is publicly maintained to foster community collaboration and iterative evaluation.

Technology Category

• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Transparent, Interpretable, Explainable ML
• Computer Vision: Interpretability, Explainability, and Transparency

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Explainable and interpretable methods for personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Mechanistic interpretability (MI) seeks to uncover how language models (LMs) implement specific behaviors, yet measuring progress in MI remains challenging. The recently released Mechanistic Interpretability Benchmark (MIB; Mueller et al., 2025) provides a standardized framework for evaluating circuit and causal variable localization. Building on this foundation, the BlackboxNLP 2025 Shared Task extends MIB into a community-wide reproducible comparison of MI techniques. The shared task features two tracks: circuit localization, which assesses methods that identify causally influential components and interactions driving model behavior, and causal variable localization, which evaluates approaches that map activations into interpretable features. With three teams spanning eight different methods, participants achieved notable gains in circuit localization using ensemble and regularization strategies for circuit discovery. With one team spanning two methods, participants achieved significant gains in causal variable localization using low-dimensional and non-linear projections to featurize activation vectors. The MIB leaderboard remains open; we encourage continued work in this standard evaluation framework to measure progress in MI research going forward.

Problem

• Evaluates methods for identifying causally influential components in language models
• Assesses approaches for mapping activations into interpretable feature representations
• Provides standardized framework for comparing mechanistic interpretability techniques

Innovation

• Ensemble and regularization strategies for circuit discovery
• Low-dimensional projections to featurize activation vectors
• Non-linear projections to featurize activation vectors