Probing the 3D Object-Level Understanding of Pre-Trained Detection Transformers

2026-08-02

AI Summary

This study investigates whether Detection Transformer (DETR) models pretrained solely with 2D supervision implicitly learn 3D spatial properties of objects. By employing both linear and nonlinear probing frameworks to recover object depth and 3D positions relative to the camera from object-level embeddings, we systematically analyze the relationship between DETR embeddings and 3D geometry across multiple model variants. Our experiments demonstrate for the first time that, without any explicit 3D supervision, object embeddings in 2D-only DETR models already encode robust and extractable 3D spatial information—significantly outperforming random baselines. These findings challenge conventional assumptions about the representational capacity of purely 2D object detection models and reveal an emergent understanding of 3D scene structure within their learned representations.

Abstract

Detection transformer models, including DETR and its extensions, learn to output a set of object-level embeddings that can be simultaneously decoded into 2D bounding boxes and class distributions. In this paper, we investigate what pre-trained 2D detection transformers understand about the 3D properties of objects. Specifically, we investigate the extent to which properties including the depth of objects from the camera and the 3D location of objects relative to the camera can be recovered from object-level embeddings using linear and non-linear probes. Across a range of detection transformer models, our results show a surprisingly strong and previously unknown ability of 2D DETR models to represent useful information about the 3D properties of objects, despite the complete lack of 3D supervision during model pre-training.

Problem

• 3D object understanding
• detection transformers
• object-level embeddings
• 3D properties
• pre-trained models

Innovation

• Detection Transformers
• 3D Object Understanding
• Probing
• Object-Level Embeddings
• DETR