EnTransformer: A Deep Generative Transformer for Multivariate Probabilistic Forecasting

2026-03-12

AI Summary

This work addresses the challenge of accurately modeling complex joint distributions in multivariate time series probabilistic forecasting by proposing EnTransformer, the first nonparametric approach that integrates the engression stochastic learning paradigm with the Transformer architecture. By injecting random noise and optimizing an energy score objective, EnTransformer directly learns the conditional predictive distribution without imposing parametric assumptions on its form, while preserving the Transformer’s capacity to capture long-range dependencies and cross-variable interactions. Evaluated on standard benchmarks—including Electricity, Traffic, and Solar—EnTransformer substantially outperforms existing methods, producing well-calibrated and coherent multivariate probabilistic forecast trajectories.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Reliable uncertainty quantification is critical in multivariate time series forecasting problems arising in domains such as energy systems and transportation networks, among many others. Although Transformer-based architectures have recently achieved strong performance for sequence modeling, most probabilistic forecasting approaches rely on restrictive parametric likelihoods or quantile-based objectives. They can struggle to capture complex joint predictive distributions across multiple correlated time series. This work proposes EnTransformer, a deep generative forecasting framework that integrates engression, a stochastic learning paradigm for modeling conditional distributions, with the expressive sequence modeling capabilities of Transformers. The proposed approach injects stochastic noise into the model representation and optimizes an energy-based scoring objective to directly learn the conditional predictive distribution without imposing parametric assumptions. This design enables EnTransformer to generate coherent multivariate forecast trajectories while preserving Transformers' capacity to effectively model long-range temporal dependencies and cross-series interactions. We evaluate our proposed EnTransformer on several widely used benchmarks for multivariate probabilistic forecasting, including Electricity, Traffic, Solar, Taxi, KDD-cup, and Wikipedia datasets. Experimental results demonstrate that EnTransformer produces well-calibrated probabilistic forecasts and consistently outperforms the benchmark models.

Problem

• multivariate probabilistic forecasting
• uncertainty quantification
• joint predictive distributions
• time series forecasting
• conditional distribution modeling

Innovation

• EnTransformer
• multivariate probabilistic forecasting
• engression
• energy-based scoring
• non-parametric conditional distribution