Persistent Sparse Autoencoders: Learning Feature Timescales in Language Models

2026-07-19

AI Summary

Standard sparse autoencoders struggle to model multi-timescale semantic features across tokens in language models. This work proposes Persistent Sparse Autoencoders (Persistent SAEs), which introduce learnable feature persistence coefficients to explicitly differentiate the temporal dynamics of fast and slow features: fast features capture local, transient signals, while slow features encode long-term thematic information. The method jointly models both transient and persistent semantics while maintaining high reconstruction fidelity. In long-context prompt injection monitoring tasks, slow features demonstrate stable causal efficacy, confirming their reliable capacity to track long-range semantic content.

Abstract

Sparse autoencoders (SAEs) decompose language model activations into sparse features, but standard SAEs encode each token independently and do not expose information that persists across a sequence. We introduce Persistent Sparse Autoencoders (Persistent SAEs), which extend standard SAEs by learning a persistence coefficient for each feature, allowing the model to learn which features should persist and for how long. Our experiments show that they retain competitive reconstruction quality while learning a spectrum of feature timescales: fast features behave as locally interpretable detectors, whereas slow features concentrate topic-level information in a persistent state. Moreover, as shown in a prompt-injection monitoring case study, slow features preserve detection signals and remain causally effective over long contexts. These results suggest that Persistent SAEs open up new opportunities for interpreting and monitoring language models through persistent semantic representations.

Problem

• Sparse Autoencoders
• Feature Timescales
• Language Models
• Persistent Representations
• Sequence Modeling

Innovation

• Persistent Sparse Autoencoders
• feature timescales
• sparse coding
• language model interpretability
• prompt-injection monitoring