Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference

2024-03-06

Neural Information Processing Systems

AI Summary

This work addresses probabilistic inference—forecasting, abduction, and intermediate-state estimation—for high-dimensional time series. We propose a probabilistic representation framework grounded in temporal contrastive learning. We provide the first theoretical proof that the learned latent states follow a Gaussian Markov chain structure, thereby reducing complex probabilistic inference to closed-form algebraic operations in a low-dimensional space: matrix inversion for abduction and linear interpolation for intermediate-state estimation. Integrating temporal contrastive learning, Gaussian graphical models, and linear-algebraic inference, our approach enables analytically tractable, efficient, and interpretable probabilistic reasoning. Empirical validation on synthetic tasks with up to 46 dimensions confirms the validity and scalability of the closed-form solutions, achieving substantial reductions in computational complexity compared to conventional sampling- or optimization-based methods.

Technology Category

• Reasoning under Uncertainty: Probabilistic Inference
• Machine Learning: Probabilistic Circuits and Graphical Models
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics

Abstract

Given time series data, how can we answer questions like"what will happen in the future?"and"how did we get here?"These sorts of probabilistic inference questions are challenging when observations are high-dimensional. In this paper, we show how these questions can have compact, closed form solutions in terms of learned representations. The key idea is to apply a variant of contrastive learning to time series data. Prior work already shows that the representations learned by contrastive learning encode a probability ratio. By extending prior work to show that the marginal distribution over representations is Gaussian, we can then prove that joint distribution of representations is also Gaussian. Taken together, these results show that representations learned via temporal contrastive learning follow a Gauss-Markov chain, a graphical model where inference (e.g., prediction, planning) over representations corresponds to inverting a low-dimensional matrix. In one special case, inferring intermediate representations will be equivalent to interpolating between the learned representations. We validate our theory using numerical simulations on tasks up to 46-dimensions.

Problem

• Solve probabilistic inference in high-dimensional time series data
• Provide compact solutions using contrastive learning representations
• Prove representations follow Gauss-Markov chain for efficient planning

Innovation

• Contrastive learning for time series data
• Gaussian joint distribution of representations
• Low-dimensional matrix inversion for inference