activation extraction

This study addresses the lack of systematic evaluation of tokenization strategies for magnetoencephalography (MEG) signals in current large-scale neuroimaging foundation models, which impacts both model performance and biological plausibility. The work presents the first comprehensive assessment of sample-level tokenization approaches in MEG foundation models, comparing learnable (a novel autoencoder-based tokenizer) and non-learnable strategies across multiple criteria: signal reconstruction, token prediction accuracy, biological plausibility, preservation of individual-specific information, and downstream task performance. Experiments on three public MEG datasets demonstrate that both approaches achieve comparable performance across most metrics and attain high reconstruction fidelity, suggesting that fixed tokenization schemes are sufficient to support the development of efficient and biologically plausible neural foundation models.

The impact of quantization on internal representations of large language models (LLMs) remains poorly understood, hindering their trustworthy deployment in resource-constrained settings. Method: We systematically investigate the effects of 4-bit and 8-bit quantization on neuron activations, contribution distributions, calibration performance, and redundancy across multiple LLM families (e.g., Llama, Qwen), employing neuron significance analysis, dead neuron detection, and attribution-based interpretability methods. Contribution/Results: Quantization induces no significant performance degradation or calibration shift; the proportion of dead neurons remains stable, and quantized LLMs retain higher neuron significance than smaller models. While sensitivity to quantization varies across architectures, overall robustness is strong. This work provides the first empirical characterization of representational robustness under quantization, revealing mechanistic insights into how LLMs preserve functional integrity post-compression. Our findings establish theoretical foundations and practical guidelines for reliable lightweight LLM compression and deployment.

Downstream probing only assesses task-relevant information in representations, failing to characterize critical properties—such as equivariance, invariance, and disentanglement—that govern interpretability and generalization; moreover, existing evaluation frameworks lack standardization, modularity, and cross-modal applicability. Method: We propose the first representation quality assessment framework that transcends downstream tasks, employing controlled factorial probe design to systematically quantify informativeness, equivariance, invariance, and disentanglement. The framework is modular, interpretable, and supports cross-modal analysis (e.g., image and speech). Contribution/Results: It establishes the first standardized, multi-dimensional semantic attribute disentanglement protocol. Experiments reveal substantial divergence in intrinsic representation properties—even among models with comparable downstream performance—enabling fine-grained representation understanding, diagnosis, and optimization. This work introduces a novel paradigm and practical toolkit for representation evaluation beyond task-specific metrics.

This work challenges the conventional assumption that final-layer representations in large language models (LLMs) are optimal, revealing instead that intermediate-layer hidden states encode richer and more robust semantic information. Method: We propose the first multidimensional representation quality evaluation framework integrating information-theoretic measures (mutual information, compression ratio), manifold geometry, and perturbation invariance—designed for cross-architectural (Transformer/SSM) and cross-modal (text/vision) validation. Contribution/Results: Evaluated on 32 text embedding benchmarks, intermediate-layer embeddings consistently outperform final-layer counterparts by an average of 4.2%, demonstrating both statistical consistency and strong generalization across tasks and architectures. This study provides the first empirical evidence establishing the superiority of intermediate-layer representations, thereby introducing a new paradigm for efficient representation extraction, model compression, and interpretability research.

This work addresses the interpretability challenge posed by distributed representations in neural networks. We propose Activation Spectrum Analysis (ActSpec), a novel method that models layer-wise activation patterns as pseudo-Boolean functions and quantifies the joint contribution of neuron subsets to network outputs via their Fourier spectra. Our approach constitutes the first systematic application of pseudo-Boolean Fourier analysis to representation interpretability. To identify high-contribution, low-redundancy Fourier coefficients efficiently, we design a constrained combinatorial optimization algorithm extending the Goldreich–Levin algorithm. Empirical evaluation on synthetic data, an MNIST classifier, and a Transformer-based sentiment analysis model demonstrates that ActSpec significantly outperforms existing methods. It provides quantitative insights into the distributed nature of representations and uncovers critical cooperative mechanisms among neurons. By bridging harmonic analysis and neural interpretability, ActSpec establishes a new paradigm for explaining distributed neural representations.

Current approaches to mechanistic interpretability lack a unified, structured representation, hindering reusability and queryability. This work proposes the Manifestation Units protocol, which standardizes interpretability findings of neural network components into evaluable representational bottlenecks through typed tuples (E, S, R, D, G) and attention-head primitives (T). The protocol enables cross-architecture generality, hybrid natural language retrieval, and causal validation. Empirical evaluations across multiple models demonstrate significant advantages over unstructured baselines: it verifies that CNN filters satisfy causal necessity and sufficiency, successfully reproduces IOI circuit membership, and identifies the core irreducible subspace S+R.

This study investigates differences in neural activation patterns across diverse cognitive tasks among various large language model architectures. Employing a unified framework, the authors systematically analyze final-layer activations, attention entropy, and sparsity across six prominent architectures on twelve task categories, yielding 144 task–model combinations. The work reveals, for the first time, a fundamental distinction between encoder- and decoder-based models in their task-processing mechanisms: mathematical reasoning consistently elicits the highest attention entropy, while decoder-only models exhibit significantly greater activation sparsity. These findings demonstrate the joint influence of architecture type and task category on internal representations, providing empirical guidance for model selection and optimization in large-scale data scenarios.

Large language models exhibit poor performance on simple symbolic tasks such as character counting, yet the underlying failure mechanisms remain unclear. This work investigates character counting as a probe task and employs interpretability techniques—including linear probing classifiers, activation patching, Logit Lens analysis, and attention head tracing—to reveal that early network layers correctly encode character information, but later MLP layers contain “negative circuits” that actively suppress the correct signal, leading to erroneous outputs. These findings demonstrate that failures in symbolic reasoning stem not from absent representations but from structured internal interference. Moreover, the study identifies, for the first time, a competitive decoding mechanism within the forward pass of large language models, which explains why scaling and instruction tuning can paradoxically exacerbate such errors.

This work addresses the limitations of existing orthogonal transformation-based quantization methods for large language models (LLMs), which rely on static one-to-one constraints and struggle to accommodate the dynamic activation distribution differences across tokens in multimodal or diffusion models. To overcome this, we propose FreeAct, a novel framework that leverages the rank-deficient nature of activations to construct a solution space beyond inverse matrices, thereby decoupling activation and weight transformations. FreeAct enables dynamic, token-aware orthogonal quantization by assigning dedicated activation transformation matrices to different token types while maintaining a unified weight transformation. Experiments on multimodal and diffusion LLMs demonstrate that FreeAct significantly outperforms existing approaches, achieving performance gains of up to 5.3%.

This work identifies a "Massively Emergent activation layer" (ME Layer) in large language models, where large-scale activations first emerge and propagate through residual connections, leading to rigid deep hidden representations and attention sink phenomena. By integrating RMSNorm and feed-forward network analysis, residual path tracing, and hidden state perturbations, the study consistently locates the ME Layer across multiple model families and elucidates its underlying mechanisms. Building on this insight, the authors propose a training-free intervention strategy that significantly enhances model performance on instruction-following and mathematical reasoning tasks while effectively mitigating attention sink issues.