On the Efficacy of Self-Supervised Point Cloud Encoders for Efficient 3D Large Language Models

2026-07-31

AI Summary

This work addresses the reliance of existing 3D language models on costly multimodal point cloud encoders, which hinders both research and deployment. The authors systematically evaluate the effectiveness of low-cost, self-supervised point cloud encoders—such as PCP-MAE and Point-MAE—within the MiniGPT-3D framework, integrating MaskTransformer and PointTransformer architectures while exploring diverse pretraining objectives and fine-tuning strategies. Their experiments reveal a strong interaction between self-supervised pretraining objectives and encoder architecture. Notably, randomly initialized encoders trained end-to-end achieve performance approaching that of pretrained counterparts, reaching 52.50% open-vocabulary accuracy, with the best configuration attaining 59.00%. However, purely geometric encoders exhibit fundamental limitations on closed-set classification tasks like ModelNet40, achieving only 13–18% accuracy—significantly below the 62% of multimodal baselines.

Abstract

3D point cloud-language models (3D-LLMs) enable 3D understanding by pairing point cloud encoders with large language models, but existing methods rely on costly multi-modal encoders (e.g., ULIP-2) that require image-text-point cloud alignment on 8x A100-scale compute, creating high barriers for research and deployment. In this work, we systematically investigate whether low-cost self-supervised point cloud encoders, specifically PCP-MAE and Point-MAE, can serve as effective alternatives. Using MiniGPT-3D as our testbed, we evaluate 7 encoder initialization/pre-training setups (1 multi-modal baseline, 5 self-supervised, 1 random init) under frozen and unfrozen fine-tuning (12 total groups), across 2 architectures (MaskTransformer, PointTransformer), 3 objectives (PCP-MAE, Point-MAE, random init), and 2 datasets (Objaverse 660K, ShapeNet55-34 approximately 50K). Our experiments reveal three key findings: (1) The four-stage MiniGPT-3D pipeline can effectively train a 3D encoder from random initialization: an end-to-end trained random init encoder reaches 52.50% open-vocabulary accuracy and 44.45 captioning score, approaching top pre-trained variants; (2) Architecture and pre-training objective show strong crossover interaction: PCP-MAE + MaskTransformer achieves 59.00% accuracy (best self-supervised), while Point-MAE + MaskTransformer drops to 46.50%, with the pattern reversed for PointTransformer; (3) Closed-set ModelNet40 classification remains a core weakness of purely geometric encoders, reaching only ~13-18% accuracy vs. ~62% for the multi-modal baseline, even after end-to-end fine-tuning. Our results offer practical guidelines for cost-effective 3D-LLM design and reveal interaction patterns between self-supervised objectives and encoder architectures.

Problem

• 3D point cloud
• large language models
• self-supervised learning
• multimodal encoders
• computational cost

Innovation

• self-supervised learning
• point cloud encoder
• 3D large language model
• PCP-MAE
• architecture-pretraining interaction