Sequence Modeling with Spectral Mean Flows

2025-10-17

AI Summary

This work addresses the challenging problem of modeling highly nonlinear, stochastic sequential state dynamics. We propose a novel operator-theoretic hidden Markov framework. Methodologically, (1) latent states are represented via mean-embedding tensors in a Hilbert space, and a scalable tensor network architecture is constructed using spectral decomposition; (2) the maximum mean discrepancy (MMD) gradient flow is generalized to time-varying reproducing kernel Hilbert spaces (RKHS) and coupled with the continuity equation, enabling simulation-free training and efficient sampling; (3) the spectral mean flow algorithm unifies flow matching and operator-theoretic dynamical modeling. Evaluated on multiple benchmark time-series datasets, the method achieves state-of-the-art performance in both predictive accuracy and sampling speed, demonstrating superior modeling expressivity and computational efficiency.

Technology Category

• Machine Learning: Matrix & Tensor Methods
• Reasoning under Uncertainty: Stochastic Optimization
• Planning, Routing, and Scheduling: Planning with Markov Models (MDPs, POMDPs)

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

A key question in sequence modeling with neural networks is how to represent and learn highly nonlinear and probabilistic state dynamics. Operator theory views such dynamics as linear maps on Hilbert spaces containing mean embedding vectors of distributions, offering an appealing but currently overlooked perspective. We propose a new approach to sequence modeling based on an operator-theoretic view of a hidden Markov model (HMM). Instead of materializing stochastic recurrence, we embed the full sequence distribution as a tensor in the product Hilbert space. A generative process is then defined as maximum mean discrepancy (MMD) gradient flow in the space of sequences. To overcome challenges with large tensors and slow sampling convergence, we introduce spectral mean flows, a novel tractable algorithm integrating two core concepts. First, we propose a new neural architecture by leveraging spectral decomposition of linear operators to derive a scalable tensor network decomposition of sequence mean embeddings. Second, we extend MMD gradient flows to time-dependent Hilbert spaces and connect them to flow matching via the continuity equation, enabling simulation-free learning and faster sampling. We demonstrate competitive results on a range of time-series modeling datasets. Code is available at https://github.com/jw9730/spectral-mean-flow.

Problem

• Modeling highly nonlinear probabilistic state dynamics in sequences
• Overcoming tractability challenges in sequence distribution embeddings
• Enabling scalable simulation-free learning for time-series modeling

Innovation

• Spectral decomposition enables scalable tensor network embeddings
• MMD gradient flows extend to time-dependent Hilbert spaces
• Simulation-free learning via continuity equation flow matching