Few-Shot Open-Vocabulary Remote Sensing Segmentation via Textual Inversion

2026-07-28

AI Summary

This work addresses the significant performance drop of open-vocabulary remote sensing image segmentation on unseen categories, which stems from the misalignment between generic textual queries and the domain-specific visual-semantic space of remote sensing imagery. To bridge this gap, the study introduces textual inversion into this task for the first time, learning category-specific text embeddings from only a few examples to replace original class names. This enables purely text-driven few-shot inference under a frozen vision-language model, without modifying the model architecture or incorporating visual prompts. Evaluated on representative benchmarks, the proposed method boosts the mean Intersection-over-Union (IoU) for affected categories from 3.9 to 39.4 and consistently outperforms existing few-shot approaches that rely on visual prompting across eight remote sensing datasets.

Abstract

Open-vocabulary segmentation labels arbitrary categories from a text query without per-class training, yet on remote sensing imagery it underperforms on categories it handles reliably elsewhere. We find that much of this gap traces to the text query rather than to the segmentation model. Because these models are not specialized for overhead imagery, the class name that serves as the query is often a weak address into the vision-language embedding space. We show that a better name repairs part of the gap, while the remaining failures call for an address that the tested natural-language rephrasings do not provide. We recover that address from a few examples through textual inversion on a frozen model, keeping inference text only. On a representative benchmark this raises the mean intersection over union on the affected categories from 3.9 to 39.4, and across eight remote sensing datasets it improves over few-shot methods that instead inject visual prompts at inference.

Problem

• open-vocabulary segmentation
• remote sensing
• few-shot learning
• textual query
• vision-language embedding

Innovation

• textual inversion
• open-vocabulary segmentation
• few-shot learning
• remote sensing
• vision-language embedding