Two-Layer Linear Auto-Regressive Models Estimate Latent States

2026-06-10

AI Summary

This work investigates how a two-layer linear autoregressive model can learn latent state representations from data generated by partially observed linear dynamical systems, without access to explicit system knowledge, via empirical risk minimization. Theoretical analysis reveals that the model’s implicit representation aligns with the Kalman filter’s state estimates up to a similarity transformation. Furthermore, the optimization landscape exhibits a benign structure: all stationary points are either strict saddle points or globally optimal. The study also provides finite-sample statistical guarantees for prediction error, parameter recovery, and state estimation accuracy. These results elucidate the intrinsic mechanism by which autoregressive models—despite lacking prior system information—can effectively approximate the optimal state estimator.

Abstract

Auto-regressive models have emerged as powerful tools for sequential data, from language to video. Understanding how and why these models learn latent representations remains an open theoretical question. In this work, we demonstrate that when trained by empirical risk minimization on data from partially observed linear dynamical systems, two-layer linear auto-regressive models naturally learn to approximate Kalman filtering. In particular, we show that the learned hidden representation coincides, up to a similarity transformation, with the state estimates produced by the optimal (Kalman) filter, even though the model has no explicit knowledge of the underlying dynamics or state. The result follows from three main insights. First, we establish that the Kalman filter is well approximated by an auto-regressive model with bounded truncation error. Second, we show that despite non-convexity, the two-layer optimization landscape is benign, i.e., all stationary points are either strict saddles or global minima. Finally, as our main contributions, we provide finite-sample guarantees on prediction error, parameter estimation error, and latent state recovery. Numerical simulations support the theoretical results and demonstrate that the latent representations of auto-regressive models recover state estimates.

Problem

• auto-regressive models
• latent states
• Kalman filtering
• linear dynamical systems
• representation learning

Innovation

• linear auto-regressive models
• Kalman filtering
• latent state estimation
• benign optimization landscape
• finite-sample guarantees