MeshFM: 2D Features Are All You Need for 3D Shape Understanding

2026-07-29

AI Summary

This work addresses the challenge of efficiently learning general-purpose 3D shape representations without access to 3D annotations. It proposes a feed-forward framework that distills semantic knowledge from 2D foundation models into 3D space via a two-stage training strategy, directly regressing a 3D feature field without requiring test-time optimization or any form of 3D supervision. Relying solely on multi-view 2D images for training, the method exhibits strong robustness to extreme object rotations and achieves performance on par with approaches that leverage explicit 3D labels across diverse downstream tasks—including part segmentation, dense correspondence, and mesh deformation—without the need for task-specific fine-tuning.

Abstract

We present MeshFM, an efficient feedforward framework for extracting rich features from 3D inputs. Our method distills 2D features from visual foundation models into 3D. We train a feedforward network to directly predict 3D features without requiring optimization during inference. The approach utilizes a two-stage training strategy. First, we optimize a feature field in 3D using only 2D feature supervision. Second, we train a network to regress this feature field. The entire procedure requires no 3D annotation, instead relying on the powerful information in 2D foundation models. We demonstrate that our learned features can be immediately applied to downstream tasks, including part segmentation, dense correspondence, and mesh deformation. Extensive experiments show that MeshFM, trained solely with 2D supervision, performs on par with methods trained explicitly with 3D supervision, even without task-specific fine-tuning. Moreover, our model is trained to be robust to extreme rotations of the input objects. Project page: https://threedle.github.io/MeshFM/

Problem

• 3D shape understanding
• 2D supervision
• feature distillation
• mesh representation
• foundation models

Innovation

• 3D shape understanding
• 2D-to-3D feature distillation
• feedforward 3D feature extraction
• foundation model supervision
• rotation robustness