GAPrompt: Geometry-Aware Point Cloud Prompt for 3D Vision Model

2025-05-07

AI Summary

To address the insufficient geometric modeling capability in parameter-efficient fine-tuning (PEFT) of point cloud pre-trained models, this paper proposes a geometry-aware point cloud prompting method. Our approach jointly models geometry at both input and feature levels: (i) point-wise prompt embeddings capture local structural patterns; (ii) an explicit shape-guided Point Shift Prompter enhances global shape awareness via learnable point offsets; and (iii) a cross-layer prompt propagation mechanism enables multi-granularity geometric information fusion. To our knowledge, this is the first multi-granularity prompting paradigm explicitly designed for point cloud geometry, overcoming the limitations of conventional token-level prompting in 3D spatial modeling. Extensive experiments on multiple point cloud benchmarks demonstrate that our method achieves performance on par with full fine-tuning while significantly outperforming state-of-the-art PEFT approaches—using only 2.19% trainable parameters—and substantially reducing computational and memory overhead.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Machine Learning: Hardware-aware ML
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Pre-trained 3D vision models have gained significant attention for their promising performance on point cloud data. However, fully fine-tuning these models for downstream tasks is computationally expensive and storage-intensive. Existing parameter-efficient fine-tuning (PEFT) approaches, which focus primarily on input token prompting, struggle to achieve competitive performance due to their limited ability to capture the geometric information inherent in point clouds. To address this challenge, we propose a novel Geometry-Aware Point Cloud Prompt (GAPrompt) that leverages geometric cues to enhance the adaptability of 3D vision models. First, we introduce a Point Prompt that serves as an auxiliary input alongside the original point cloud, explicitly guiding the model to capture fine-grained geometric details. Additionally, we present a Point Shift Prompter designed to extract global shape information from the point cloud, enabling instance-specific geometric adjustments at the input level. Moreover, our proposed Prompt Propagation mechanism incorporates the shape information into the model's feature extraction process, further strengthening its ability to capture essential geometric characteristics. Extensive experiments demonstrate that GAPrompt significantly outperforms state-of-the-art PEFT methods and achieves competitive results compared to full fine-tuning on various benchmarks, while utilizing only 2.19% of trainable parameters. Our code is available at https://github.com/zhoujiahuan1991/ICML2025-VGP.

Problem

• Improves 3D model fine-tuning with geometric-aware prompts
• Reduces computational cost of adapting pre-trained 3D models
• Enhances geometric detail capture in point cloud data

Innovation

• Geometry-Aware Point Cloud Prompt enhances 3D models
• Point Shift Prompter extracts global shape information
• Prompt Propagation integrates shape into feature extraction