activation analysis

This work investigates the internal inference mechanisms and parameter sensitivity of convolutional neural networks (CNNs) under concurrent multiple image augmentations. Method: We propose a controlled–comparative interpretability framework featuring a novel activation variance decomposition technique under multi-parameter input perturbations, integrating Sobol indices and Shapley values to quantify individual augmentation contributions to neuron activations and generate guidance-aware activation masks. Additionally, we introduce a one-class sensitivity analysis that identifies critical sensitive neurons via prediction deviation, enabling fine-grained sensitivity localization and visualization of cross-layer/channel functional specialization. Contribution/Results: Our approach significantly enhances CNN interpretability with respect to augmentation operations and establishes a cross-domain methodological bridge between AI interpretability research and computational modeling of biological neural systems.

This work investigates the degradation of intermediate-layer representations in Vision Transformers (ViTs) under out-of-distribution (OOD) conditions and identifies optimal probing locations. Through large-scale linear probing experiments, the authors systematically evaluate the representational capacity of different layers and modules in pretrained ViTs across varying degrees of distribution shift. They find that performance deterioration in deeper layers is primarily attributable to distributional shifts. Notably, at the module granularity, the internal activations of the feedforward network and the normalized outputs of multi-head attention emerge as optimal probing points under strong and weak distribution shifts, respectively—challenging the conventional practice of probing only block outputs. Extensive validation on multiple image classification benchmarks demonstrates that this probing strategy significantly improves downstream OOD performance.

This work addresses the limited trustworthiness of Transformer models in high-stakes applications, which stems from insufficient understanding of their internal decision-making mechanisms. To bridge this gap, we propose a mechanistic interpretability approach based on targeted interventions on attention heads, integrating causal analysis with neural circuit probing to systematically uncover the model’s decision processes and underlying cognitive mechanisms. Our method substantially enhances the interpretability of Transformer internals and offers an innovative pathway toward the design and control of highly reliable AI systems, while also enabling the discovery of novel scientific insights encoded within these models.

Traditional black-box AI evaluation relies solely on output probabilities, failing to model human real-time cognitive processing. Method: This study investigates whether layer-wise temporal dynamics during Transformer forward propagation can capture such processing, using layer-wise activation analysis and inter-layer time-series modeling, integrated with multimodal neurobehavioral data (eye-tracking, EEG, reading times) across five cross-domain regression and incremental prediction tasks in language and vision. Contribution/Results: We provide the first systematic evidence that pretrained Transformers (e.g., BERT, ViT) implicitly acquire cognition-relevant representational structures; their internal computational dynamics significantly improve prediction of human real-time processing metrics (p < 0.001), with robust cross-modal and cross-task consistency. These findings support treating AI models as interpretable “processing” models—offering a novel paradigm bridging artificial intelligence and cognitive science.

This study investigates the mechanistic degradation and reversibility of language models under toxic data fine-tuning. Toxic fine-tuning induces model corruption, yet its underlying neural mechanisms and potential for recovery remain poorly understood. Method: Leveraging causal tracing and circuit localization—key techniques from mechanistic interpretability—alongside task-specific fine-tuning and clean-data reverse retraining, we conduct controlled ablation and reconstruction experiments. Results: We establish, for the first time, that corruption exhibits *circuit-level specificity*: only critical computational pathways are selectively impaired, while peripheral circuits remain intact. Crucially, we demonstrate *neuroplastic-like recoverability*: clean-data retraining reconstructs original functional mechanisms with >89% restoration fidelity; this recovery generalizes across fine-tuning epochs. Contribution: Our work identifies precise circuit-level localization principles governing corruption and empirically validates the reversibility of mechanistic damage—providing both theoretical foundations and actionable strategies for robust alignment and trustworthy fine-tuning.

Existing neuron-level interpretability methods are often task-specific, require retraining, or are merely descriptive, making systematic evaluation of Transformer internal robustness challenging. This work proposes SYNAPSE, a framework that enables training-free, cross-architecture, and cross-domain neuron analysis and perturbation for the first time. By extracting [CLS] representations from each layer, SYNAPSE trains lightweight linear probes to rank neurons globally and per class, and applies structured perturbations during inference via forward hooks. Experiments reveal that task-relevant information is encoded by broadly overlapping and functionally redundant neuron subsets, alongside category-asymmetric specialization patterns. Notably, minimal perturbations can drastically alter model predictions, exposing inherent vulnerabilities and offering new insights into the internal mechanisms of Transformers.

This work proposes a unified framework based on continuous depth-token field theory to systematically model the effects of intervention operations—such as activation patching—on Transformer behavior. Treating the residual stream as a depth-token field, patching interventions are formalized as localized source insertions. By integrating first-order sensitivity analysis, empirical Green’s functions, and adjoint variational methods, the framework characterizes the propagation mechanisms of intervention-induced responses. It is the first to describe patching experiments using the language of field theory, revealing anisotropic response patterns across both depth and token dimensions and accurately predicting local linear responses. Empirical validation demonstrates the method’s effectiveness in prompt-induced behavioral transfer, offering a novel paradigm for interpretability analysis in deep learning models.

This study investigates how representational structures across modules—feedforward encoder, recurrent module, and readout module—of neural foundation models differ and affect their alignment with biological visual systems. We propose a physiology-inspired manifold analysis framework, introducing for the first time in foundation model interpretability the joint modeling of encoding manifolds (neurons → responses) and decoding manifolds (stimuli → activity), integrating manifold learning, parametric response modeling, and spatiotemporal pattern decomposition. Our results reveal: (1) fundamental geometric heterogeneity among the three modules’ representational manifolds; (2) the recurrent module enhances discriminability by dynamically “repelling” temporal patterns in latent space; and (3) while the readout module achieves high biological fidelity, its dedicated feature-map mechanism deviates from neurobiological plausibility. This work establishes a novel analytical paradigm for model–brain alignment and provides structured, geometry-aware interpretability grounded in neural computation principles.

This work addresses the limited interpretability of heterogeneous attention mechanisms—such as co-attention—in multimodal or multi-source information fusion, where existing approaches struggle to elucidate their internal workings. To bridge this gap, the paper introduces the first general-purpose interpretability framework tailored specifically for heterogeneous attention. By integrating attention analysis, semantic interpretation, and logical reasoning, the proposed method establishes a unified analytical paradigm. The framework is successfully applied to representative Transformer-based models, enabling in-depth semantic and logical dissection of heterogeneous attention mechanisms. Experimental results demonstrate its broad applicability and practical utility across diverse architectures, offering new insights into how such attention modules process and integrate heterogeneous inputs.

This study investigates the reverse decoding of semantic attributes—such as valence, arousal, and dominance—of original stimuli from synthetic brain activity signals. For the first time, a large language model (LLM) is employed as a controllable stimulus generator, integrated with the TRIBEv2 brain foundation model and simulation-based inference to establish a probabilistic mapping from predicted neural representations to latent stimulus parameters. The proposed approach not only successfully recovers the semantic properties of the stimuli, thereby validating the fidelity of the underlying brain encoding model, but also demonstrates the novel potential of LLMs in simulating neuroscientific experiments and enabling inverse design in neuroscience.