🤖 AI Summary
This work addresses the fundamental challenge that 2D vision models cannot be directly applied to irregular and sparse 3D data such as point clouds and meshes. To this end, it proposes the first unified taxonomy encompassing data representations, architectural designs, and hybrid strategies. Existing approaches are systematically categorized into three paradigms: projection-based data-centric methods, architecture-centric techniques leveraging native 3D structures, and hybrid approaches combining both. The study provides a thorough analysis of the trade-offs among computational complexity, reliance on pretraining, and preservation of geometric inductive biases. By integrating pathways including transfer from 2D CNNs or Vision Transformers, native 3D network design, multimodal fusion, and self-supervised learning, this work offers a systematic roadmap for advancing 3D foundation models, geometric self-supervised learning, and multimodal representation learning.
📝 Abstract
The remarkable success of Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs) in 2D vision has spurred significant research in extending these architectures to the complex domain of 3D analysis. Yet, a core challenge arises from a fundamental dichotomy between the regular, dense grids of 2D images and the irregular, sparse nature of 3D data such as point clouds and meshes. This survey provides a comprehensive review and a unified taxonomy of adaptation strategies that bridge this gap, classifying them into three families: (1) Data-centric methods that project 3D data into 2D formats to leverage off-the-shelf 2D models, (2) Architecture-centric methods that design intrinsic 3D networks, and (3) Hybrid methods, which synergistically combine the two modeling paradigms to benefit from both rich visual priors of large 2D datasets and explicit geometric reasoning of 3D models. Through this framework, we qualitatively analyze the fundamental trade-offs between these families concerning computational complexity, reliance on large-scale pre-training, and the preservation of geometric inductive biases. We discuss key open challenges and outline promising future research directions, including the development of 3D foundation models, advancements in self-supervised learning (SSL) for geometric data, and the deeper integration of multi-modal signals.