PULSE: Identifying Demonstration-Utility Features with Sparse Autoencoders

2026-09-26

AI Summary

This study addresses the limitation of in-context learning example selection methods that rely on external similarity while neglecting internal model feature signals. To this end, it proposes PULSE, a framework that leverages sparse autoencoders (SAEs) to identify internal features associated with example utility, thereby guiding retrieval and ranking. The core innovation lies in aligning SAE activation differences with utility variations to pinpoint critical internal features—a first in the literature. By integrating zero-shot relative utility measurement, feature relevance masking, and controlled full-set ranking techniques, PULSE achieves a paradigm shift from external similarity to internal mechanistic alignment. Empirical evaluations demonstrate that the proposed approach improves accuracy by 2–3 points on classification benchmarks, BLEU-4 scores by 0.6–0.9 points on generation tasks, and exact match by 3.2 points on reasoning benchmarks.

Abstract

In-context learning is highly sensitive to demonstration choice, yet most methods select demonstrations using external query-demonstration similarity. Such criteria can miss model-specific signals: Similar demonstrations may activate different internal features and downstream behaviors. We introduce PULSE (Paired Utility Localization over Sparse Encodings), an SAE-based framework for identifying model-internal features associated with demonstration utility and using them for demonstration selection. Using a small labeled discovery set, PULSE samples candidate demonstration sets, measures their zero-shot-relative utility under the target model, and scores SAE features by how their activation differences align with utility differences. The top positive and negative coordinates form a sparse utility-localization vector. We use this vector in two complementary ways: as a signed score for controlled complete-set ranking, and as PULSE-Retriever, which converts its magnitude into a feature-relevance mask for scalable pool-scale retrieval. Across classification, generation, and reasoning benchmarks, PULSE-Retriever improves over the strongest baseline by 2-3 accuracy points, 0.6-0.9 BLEU-4, and 3.2 exact-match points, respectively, while controlled ranking validates the identified features encode a predictive set-level utility signal. Feature inspection and cross-dataset experiments suggest that the identified features capture task-relevant, dataset-conditioned patterns, yet retain utility signals that partially transfer across datasets. Our code is available at https://github.com/aohenuo/PULSE.

Problem

• in-context learning
• demonstration selection
• sparse autoencoders
• demonstration utility
• model-internal features

Innovation

• Sparse Autoencoders
• In-context Learning
• Demonstration Selection
• Feature Utility Localization
• PULSE-Retriever