🤖 AI Summary
This work addresses the tendency of large language models to rely on time-specific knowledge in prediction tasks, which induces look-ahead bias. By employing sparse autoencoders to dissect internal model representations, the authors identify interpretable features associated with temporal awareness and, for the first time, apply targeted causal interventions to modulate these features. Experimental results demonstrate that enhancing temporal-awareness features significantly mitigates look-ahead bias, whereas direct intervention on bias-related features proves ineffective, underscoring the pivotal role of temporal perception in improving predictive generalization. The proposed approach effectively steers the model toward greater reliance on historical information while preserving its general reasoning capabilities.
📝 Abstract
Successful forecasting involves identifying patterns between historical and future states of the world which generalize to future observations. We apply LLMs to a variety of forecasting tasks and inspect their internal states using sparse autoencoders to understand whether they appear to rely on time-specific pieces of knowledge versus generalizable patterns. Our analyses identify features associated with both time-aware reasoning and look-ahead-biased reasoning. We then apply the LLMs to an entirely different domain and intervene on these features. We find that amplifying time-awareness features substantially reduces look-ahead bias on forecasting prompts while preserving general reasoning performance. In contrast, steering the candidate look-ahead-bias features does not produce an effect. These results suggest that interpretable temporal features can be used to causally shift LLMs toward more historically grounded reasoning.