Filling Before Advancing: Capability-Gap-Driven Post-Training for Scenario-Specialized Remote Sensing MLLMs

2026-07-24

AI Summary

This work addresses the scarcity of high-quality annotated data and insufficient capability coverage of remote sensing multimodal large language models in specialized scenarios such as coastal ports. To tackle these challenges, the authors propose a staged post-training framework termed “Fill-before-Advance” (FBA), which leverages a three-tier supervised dataset—CPRS—to sequentially perform remote sensing semantic alignment, cross-scenario domain bridging, and evidence-driven fine-tuning. This approach introduces a novel two-stage paradigm driven by capability gaps, explicitly distinguishing between foundational capability development and scenario-specific refinement. Additionally, the study constructs HarborEval, the first multi-source remote sensing evaluation benchmark dedicated to coastal ports. Experiments demonstrate that models trained with FBA achieve HarborEval scores of 70.29 and 83.37 on LLaVA-v1.5 and Qwen3-VL, respectively, significantly outperforming baseline methods such as Direct-SFT.

Abstract

Remote sensing multimodal large language models (RS-MLLMs) have improved general aerial-image understanding. However, Earth observation applications require fine-grained scenario specialization, constrained by scarce high-quality scenario data and incomplete capability coverage. We formulate this adaptation as a capability-gap-driven post-training problem and propose filling before advancing (FBA). Rather than relying on single-stage supervised fine-tuning (SFT) over target-domain samples, FBA first fills prerequisite capability gaps before advancing toward scenario specialization. We instantiate FBA for coastal harbor understanding, a representative multi-source scenario, by constructing CPRS (Coastal-Port Remote Sensing), a three-layer supervision dataset coupled with three ordered stages: (1) RS semantic anchoring for overhead-view visual-language alignment; (2) domain-bridge convergence for shared RS priors across target and bridging scenarios under different modalities; and (3) evidence-grounded scenario tuning for downstream performance. We construct HarborEval, an eight-track diagnostic benchmark covering perception, spatial understanding, robustness, and generation. Under comparable training budgets, HarborEval increases from 57.95 with Direct-SFT to 70.29 with FBA on LLaVA-v1.5, and from 81.09 to 83.37 on Qwen3-VL. FBA also outperforms Collapsed-SFT and leads on harbor-related VRSBench/RSVQA subsets and OpenEval. Stage-wise and role-replacement analyses validate progressive gap filling and stage-specific roles. Public examples and release updates for CPRS, HarborEval, code, and trained weights are available at https://github.com/Z0ngL1ng/filling-before-advancing.

Problem

• remote sensing
• multimodal large language models
• scenario specialization
• capability gap
• post-training

Innovation

• capability-gap-driven post-training
• filling before advancing
• scenario-specialized RS-MLLMs
• multi-stage supervision
• coastal harbor understanding