Co-Training Vision Language Models for Remote Sensing Multi-task Learning

2025-11-26

AI Summary

To address challenges in remote sensing multi-task learning—including complex environmental conditions, scale variability, and computational difficulty in processing ultra-high-resolution (UHR) imagery—this paper introduces RS-VLM, the first unified vision-language model specifically designed for remote sensing. Methodologically, we propose a dynamic-resolution input mechanism and a Zoom-in Chain inference paradigm; construct the LRS-VQA-Zoom benchmark dataset and a novel evaluation protocol; and integrate online data weighting with multi-task prompt learning. Our contributions are threefold: (1) enabling text-driven unified modeling across image understanding, localization, and UHR reasoning; (2) achieving state-of-the-art performance on remote sensing visual question answering (VQA), object detection, and semantic segmentation—matching or surpassing task-specific expert models; and (3) open-sourcing all code, models, and datasets to foster fair benchmarking and practical deployment of RS-VLMs.

Technology Category

• Computer Vision: Remote Sensing / Geospatial AI
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

With Transformers achieving outstanding performance on individual remote sensing (RS) tasks, we are now approaching the realization of a unified model that excels across multiple tasks through multi-task learning (MTL). Compared to single-task approaches, MTL methods offer improved generalization, enhanced scalability, and greater practical applicability. Recently, vision language models (VLMs) have achieved promising results in RS image understanding, grounding, and ultra-high-resolution (UHR) image reasoning, respectively. Moreover, the unified text-based interface demonstrates significant potential for MTL. Hence, in this work, we present RSCoVLM, a simple yet flexible VLM baseline for RS MTL. Firstly, we create the data curation engine, including data acquisition, offline processing and integrating, as well as online loading and weighting. This data engine effectively addresses complex RS data enviroment and generates flexible vision-language conversations. Furthermore, we propose a unified dynamic-resolution strategy to address the diverse image scales inherent in RS imagery. For UHR images, we introduce the Zoom-in Chain mechanism together with its corresponding dataset, LRS-VQA-Zoom. The strategies are flexible and effectively mitigate the computational burdens. Additionally, we significantly enhance the model's object detection capability and propose a novel evaluation protocol that ensures fair comparison between VLMs and conventional detection models. Extensive experiments demonstrate that RSCoVLM achieves state-of-the-art performance across diverse tasks, outperforming existing RS VLMs and even rivaling specialized expert models. All the training and evaluating tools, model weights, and datasets have been fully open-sourced to support reproducibility. We expect that this baseline will promote further progress toward general-purpose RS models.

Problem

• Developing unified vision-language models for multiple remote sensing tasks
• Addressing computational challenges of ultra-high-resolution remote sensing imagery
• Creating flexible data processing for complex remote sensing environments

Innovation

• Co-training vision language models for remote sensing
• Unified dynamic-resolution strategy for diverse image scales
• Zoom-in Chain mechanism for ultra-high-resolution images