MinkowskiPE: Minkowski Positional Encoding for Spatiotemporal Perception

2026-09-27

AI Summary

This study addresses the inherent difficulty of reconciling physical priors with learning flexibility in spatiotemporal coupled modeling by proposing Minkowski positional encoding. The method leverages Lorentz transformations to jointly parameterize query-key features alongside spatiotemporal coordinates, thereby explicitly modeling spatiotemporal geometric relationships. This formulation ensures that attention scores depend solely on relative displacements, preserving global translation invariance while remaining compatible with efficient attention mechanisms. Experimental results demonstrate that the proposed model achieves state-of-the-art performance on both molecular dynamics simulations and KTH video prediction tasks. Notably, it reduces the parameter count by approximately 90% while decreasing prediction error by 9.9%, highlighting its exceptional efficiency and effectiveness in capturing complex spatiotemporal dynamics.

Abstract

Modeling spatiotemporal coupling is a key challenge in building physical intelligence across scales, from microscopic to macroscopic. Existing models capture such structure broadly through physics-motivated dynamical formulations or learning-motivated architectures. The former provide stronger priors but may constrain flexibility, whereas the latter are more flexible but leave the spatiotemporal coupling largely implicit. We therefore seek an approach that combines flexible learning with an explicit geometric bias for jointly modeling time and space. To this end, we propose Minkowski Positional Encoding (MinkowskiPE), which uses joint temporal and spatial coordinates to parameterize Lorentz transformations applied to query and key features. With MinkowskiPE, the query-key attention score depends on position only through the relative spacetime displacement between the two tokens and is therefore invariant to global translation of the coordinates. This paradigm retains the standard dot-product attention interface and remains compatible with efficient attention implementations. We evaluate MinkowskiPE on microscopic molecular dynamics and macroscopic video prediction tasks, achieving the best results on all nine multi-trajectory molecular evaluations and reducing KTH video-prediction MSE by 9.9% relative to the best baseline while using roughly one-tenth as many parameters.

Problem

• spatiotemporal coupling
• positional encoding
• physical intelligence
• Lorentz transformations
• attention mechanism

Innovation

• Minkowski Positional Encoding
• Lorentz Transformations
• Spatiotemporal Coupling
• Translation Invariance
• Dot-Product Attention