Beyond Token Scale: Chunk-Level Sparse Autoencoders for Reliable Semantic Feature Discovery

2026-09-28

AI Summary

This study addresses the challenge that token-level Sparse Autoencoders (SAEs) struggle to extract reliable semantic features due to budget competition. To overcome this limitation, we propose chunk-level SAEs, which encode pooled activations of contiguous text chunks to disentangle observational units from cross-segment predictive influences. Furthermore, we introduce three novel architectures—Mean, Cross, and Joint-Chunk—that modify the input receptive field and prediction targets to capture high-level conceptual features. Experimental results demonstrate that the proposed approach significantly enhances performance across downstream tasks, including document retrieval, reasoning detection, and model steering. Ultimately, this method facilitates the discovery of more selective and persistent high-level semantic representations, advancing the interpretability and utility of sparse autoencoders in large language models.

Abstract

Sparse autoencoders (SAEs) expose features that help us understand and steer language models, but faithful reconstruction does not guarantee informative concepts. Token-level objectives reward lexical and formatting details alongside semantic content, all competing for a limited sparse budget. We introduce a family of chunk-level SAEs that encode mean-pooled activations over chunks, each a contiguous span of tokens: Mean-Chunk reconstructs the observed chunk, Cross-Chunk predicts an independently processed neighbor, and Joint-Chunk combines both targets. These designs separate the effect of a larger observation unit from that of predicting information shared across passages. With matched training data, chunk-level SAEs remain powerful interpretability tools while learning reliable semantic features that capture high-level concepts and respond selectively to relevant content. Their strengths are complementary: Mean-Chunk improves high-level feature discovery, reasoning detection beyond surface cues, and steering; Cross-Chunk leads document retrieval and classification transfer while producing selective, persistent features. Changing what an SAE sees and predicts yields reliable semantic features for more meaningful tasks. We demonstrate their practical value through gains across downstream tasks such as retrieval, reasoning detection, and steering.

Problem

• Sparse Autoencoders
• Semantic Feature Discovery
• Token-level Objectives
• Interpretability
• Language Models

Innovation

• Sparse Autoencoders
• Chunk-Level Encoding
• Semantic Feature Discovery
• Mechanistic Interpretability
• Mean Pooling