Auto-Prompting SAM for Weakly Supervised Landslide Extraction

2025-01-23

AI Summary

In weakly supervised landslide detection, coarse-grained image-level labels and complex landslide morphologies lead to inaccurate boundary localization. To address this without pixel-level annotations or fine-tuning of the Segment Anything Model (SAM), this paper proposes a prompt-driven segmentation framework grounded solely in prompt engineering. Its key contributions are: (1) an adaptive hybrid prompting mechanism that jointly leverages bounding boxes and center points; (2) the Adaptive Pattern Generation (APG) algorithm, which efficiently mines discriminative visual patterns from Class Activation Maps (CAMs) to generate high-quality pseudo-masks end-to-end; and (3) a streamlined inference chain—“localization network → CAM → APG → SAM”—that requires no model adaptation. Evaluated on high-resolution aerial and satellite datasets, our method achieves ≥3.0% improvement in F1-score and a 3.69% gain in IoU over state-of-the-art weakly supervised approaches.

Technology Category

• Natural Language Processing: Prompt Engineering / Prompting
• Computer Vision: Segmentation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Weakly supervised landslide extraction aims to identify landslide regions from remote sensing data using models trained with weak labels, particularly image-level labels. However, it is often challenged by the imprecise boundaries of the extracted objects due to the lack of pixel-wise supervision and the properties of landslide objects. To tackle these issues, we propose a simple yet effective method by auto-prompting the Segment Anything Model (SAM), i.e., APSAM. Instead of depending on high-quality class activation maps (CAMs) for pseudo-labeling or fine-tuning SAM, our method directly yields fine-grained segmentation masks from SAM inference through prompt engineering. Specifically, it adaptively generates hybrid prompts from the CAMs obtained by an object localization network. To provide sufficient information for SAM prompting, an adaptive prompt generation (APG) algorithm is designed to fully leverage the visual patterns of CAMs, enabling the efficient generation of pseudo-masks for landslide extraction. These informative prompts are able to identify the extent of landslide areas (box prompts) and denote the centers of landslide objects (point prompts), guiding SAM in landslide segmentation. Experimental results on high-resolution aerial and satellite datasets demonstrate the effectiveness of our method, achieving improvements of at least 3.0% in F1 score and 3.69% in IoU compared to other state-of-the-art methods. The source codes and datasets will be available at https://github.com/zxk688.

Problem

• Weak Supervision
• Landslide Detection
• Label Precision

Innovation

• APSAM
• Weakly-Supervised Learning
• Landslide Detection