Generative sequence modeling for infinite memory processes via predictive states

2026-09-29

AI Summary

This study addresses the curse of dimensionality in nonparametric estimation arising from long-range nonlinear interactions within infinite memory processes. To overcome this challenge, we propose a deep neural network-based approach for generative sequence modeling and predictive state compression. By compressing historical observations into low-dimensional predictive states, our method transcends conventional structural assumptions and enables efficient conditional distribution estimation. We theoretically establish that the estimation complexity is governed solely by the intrinsic dimensionality of the predictive state space, rather than the full observation history. Furthermore, fast convergence rates are rigorously guaranteed under mild regularity conditions. Extensive experiments comprehensively validate the effectiveness of the proposed framework, demonstrating its superiority in capturing complex temporal dependencies while maintaining computational tractability.

Abstract

We consider estimating the one-step-ahead conditional distribution of a multivariate stochastic process. Many existing approaches rely on assumptions such as finite-range memory, sparsity, or additivity, which can be poorly suited to processes with long-range nonlinear interactions. However, without such structural assumptions, nonparametric estimation is challenging due to the curse of dimensionality. To address this challenge, we introduce a new estimation approach based on the predictive states of a process, possibly with infinite-range memory. We show that our estimator achieves fast convergence rates when the past history can be compressed into a low-dimensional statistic that is sufficient for predicting the future. Specifically, we show that the statistical complexity of the estimation problem is determined by the intrinsic dimension of the predictive state space. We establish guarantees for an instantiation of our method based on deep neural network estimators, and we support these theoretical results with experiments.

Problem

• generative sequence modeling
• predictive states
• infinite memory processes
• conditional distribution estimation
• curse of dimensionality

Innovation

• predictive states
• infinite memory processes
• generative sequence modeling
• deep neural networks
• statistical complexity