🤖 AI Summary
This work addresses the challenge of predicting entirely missing objects in 3D scenes by introducing SR-JEPA, a scene-level point cloud joint-embedding predictive architecture. The method learns context-aware, queryable 3D latent states without requiring reconstruction, semantic labels, or 2D features, leveraging shape-agnostic, learnable query points anchored at object centers. As the first approach to achieve compositional predictive representations for fully absent objects, SR-JEPA employs a point-native JEPA framework with an exponential moving average target and a frozen prediction pathway. Experiments demonstrate its effectiveness, achieving a 43.13% semantic identity macro accuracy on ARKitScenes—surpassing the strongest baseline by 22.18 points—and attaining 41.15 AP on Sr3D, thereby validating the representational power and generalization capability of the learned latent states.
📝 Abstract
Joint-embedding predictive architectures learn by predicting latent representations of missing observations, yet many masked JEPAs are evaluated primarily through the encoders they produce. We ask what a trained predictive pathway itself infers when an entire entity is absent from a native 3D scene. We introduce SR-JEPA, a point-native JEPA for scene-scale point clouds whose original frozen predictive pathway can be queried at a supplied location. At evaluation, every point of one object is removed before encoding and replaced by the same shape-free 32-point query at its centroid. Training uses only self-contained 3D EMA targets: no reconstruction, semantic labels, language, or lifted 2D features. On 5,953 held-out ARKitScenes objects, the imputed latent reaches 43.13% semantic-identity macro accuracy, 22.18 points above the strongest floor. Randomizing the prediction path removes 9.78 points, while substituting matched donor context removes 21.98 points. On 8,570 Sr3D support pairs, the full latent reaches 41.15 AP; identity decoded from the missing-object latent, combined with anchor identity and geometry, reaches 39.37 AP, leaving an unresolved 1.78-point residual. These results reveal a queryable, compositional 3D predictive state: the model completes context-dependent entity content, which downstream computation combines with metric geometry.