Emergent Latent-State Computation under Stochastic Volatility

2026-07-28

AI Summary

This study investigates how sequence models can infer latent stochastic states—such as volatility—from noisy observations like financial returns. Within a multivariate stochastic volatility framework, the authors establish the first interpretability benchmark in a controlled stochastic dynamic environment to systematically evaluate diverse neural architectures, loss functions, and output head designs. Their analysis reveals that Transformers decode latent states effectively at specific structural stages, and that a simplified filtering mechanism—comprising linear projection followed by ℓ² normalization—performs well over long horizons. Furthermore, the model exhibits a two-stage internal computation: hidden layers encode the next-step latent state, while the output head maps this representation to squared return predictions. Misalignment between these stages is identified as the primary cause of performance degradation under noise-sensitive MSE training.

Abstract

Mechanistic interpretability has largely focused on language models and deterministic toy tasks. Much less is known about how sequence models internally represent latent stochastic dynamics under noisy, partially observed observations. We study this question in a controlled multivariate stochastic volatility setting, where models observe only returns while the ground-truth latent volatility state is known to the researcher. This setting provides a useful benchmark for mechanistic interpretability under partial observability: the latent state is hidden from the model but directly available for evaluation. Across architectures, losses, and output heads, we find evidence for a two-stage computation. Hidden representations encode substantial information about the next latent volatility state, and the output head maps this representation to squared return forecasts. Furthermore, in Transformers, latent-state decodability emerges at identifiable architectural stages whose location depends on the volatility period. In long-cycle regimes, this computation simplifies into an explicit latent-state filter consisting of a learned linear projection followed by $\ell^2$ normalization. Output-head replacement further shows that part of the degradation under noisy MSE training arises from readout misalignment rather than representation failure. These results suggest that stochastic volatility models provide a useful benchmark for mechanistic interpretability under noisy latent dynamics and partial observability.

Problem

• stochastic volatility
• latent dynamics
• partial observability
• mechanistic interpretability
• sequence models

Innovation

• stochastic volatility
• mechanistic interpretability
• latent-state computation
• partial observability
• Transformer dynamics