A Graph Signal Processing Perspective on Numerical Sequence Representations in LLM In-Context Learning

2026-08-03

AI Summary

This study investigates how large language models organize internal representations of numerical sequences during in-context learning. It introduces, for the first time, a systematic application of graph signal processing by treating the token graph induced by attention mechanisms as a weighted graph, with hidden states serving as node signals, and analyzes its spectral properties. The findings reveal that as context length increases, simple inputs yield globally connected graphs with smooth signals, whereas complex inputs lead to localized connectivity and stronger high-frequency components. This pattern consistently emerges across multiple model families, uncovering an intrinsic relationship between the dynamic complexity of inputs and the spectral characteristics of internal representations.

Abstract

Pretrained large language models (LLMs) have demonstrated in-context learning (ICL) capabilities for numerical inference over sequences serialized as text. Prior work has identified and characterized this form of numerical inference primarily through output-level evaluations such as prediction error. However, how numerical information is organized within LLM representations remains much less understood. To study this internal organization, we adopt a graph signal processing perspective in which attention induces a weighted graph over tokens, while token hidden states define signals on its nodes. Quantitative graph-spectral diagnostics and qualitative token-graph visualizations reveal that representations become more clearly differentiated by input dynamical complexity as context length increases. Simpler inputs produce attention-induced token graphs with stronger global connectivity and smoother, spectrally concentrated hidden-state signals, whereas more complex inputs produce more localized graphs and hidden-state signals with broader spectral support and greater high-frequency energy. Together, these findings point to systematic, context-dependent internal signatures associated with numerical ICL that are conserved across model families.

Problem

• in-context learning
• numerical sequence
• representation organization
• large language models
• graph signal processing

Innovation

• Graph Signal Processing
• In-Context Learning
• Numerical Sequence Representation
• Attention-induced Graph
• Spectral Analysis