🤖 AI Summary
Traditional Gaussian processes suffer from computational intractability in high-dimensional spatiotemporal forecasting, while Transformers lack spatial geometric priors, limiting physical interpretability and extrapolation.
Method: We propose Spatially-Aware Transformer (SAT), which embeds a learnable anisotropic covariance kernel into the self-attention mechanism, decomposing it into a physics-driven stationary prior and a data-driven non-stationary residual. We introduce “deep variogram analysis” for end-to-end inversion of spatial decay parameters and incorporate soft topological constraints to inject geostatistical inductive bias.
Contribution/Results: SAT achieves state-of-the-art performance on synthetic Gaussian random fields and real-world traffic datasets—outperforming graph neural networks in both point prediction accuracy and probabilistic calibration. It yields strictly calibrated uncertainty estimates while unifying physical interpretability with data-driven expressiveness, enabling scalable, principled spatiotemporal forecasting.
📝 Abstract
The modeling of high-dimensional spatio-temporal processes presents a fundamental dichotomy between the probabilistic rigor of classical geostatistics and the flexible, high-capacity representations of deep learning. While Gaussian processes offer theoretical consistency and exact uncertainty quantification, their prohibitive computational scaling renders them impractical for massive sensor networks. Conversely, modern transformer architectures excel at sequence modeling but inherently lack a geometric inductive bias, treating spatial sensors as permutation-invariant tokens without a native understanding of distance. In this work, we propose a spatially-informed transformer, a hybrid architecture that injects a geostatistical inductive bias directly into the self-attention mechanism via a learnable covariance kernel. By formally decomposing the attention structure into a stationary physical prior and a non-stationary data-driven residual, we impose a soft topological constraint that favors spatially proximal interactions while retaining the capacity to model complex dynamics. We demonstrate the phenomenon of ``Deep Variography'', where the network successfully recovers the true spatial decay parameters of the underlying process end-to-end via backpropagation. Extensive experiments on synthetic Gaussian random fields and real-world traffic benchmarks confirm that our method outperforms state-of-the-art graph neural networks. Furthermore, rigorous statistical validation confirms that the proposed method delivers not only superior predictive accuracy but also well-calibrated probabilistic forecasts, effectively bridging the gap between physics-aware modeling and data-driven learning.