Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

2026-07-22

AI Summary

This work addresses the challenges of language-guided small object understanding in online aerial videos, where visual detail loss and difficulty in modeling historical context hinder performance. Focusing on real-time drone flight scenarios, the study introduces DroneEyes, the first fine-grained, open-vocabulary referring segmentation dataset for aerial small objects, comprising 2,140 high-definition videos and 176,623 annotated referring-expression pairs. To tackle these challenges under limited computational resources, the authors propose SkyAnchor, a multimodal large language model that leverages semantic-aware visual token routing and a hierarchical memory bank to effectively preserve fine-grained object details and model streaming contextual information. Experimental results demonstrate that the proposed approach significantly enhances both the stability and accuracy of small object recognition in continuous aerial video streams.

Abstract

Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames. However, applying current Multimodal Large Language Models (MLLMs) to this setting raises two challenges. First, targets viewed from the air are often tiny, yet the visual compression in existing MLLMs treats all regions equally and discards their fine-grained details. Second, understanding a continuous stream requires past-frame context, yet retaining the entire history is infeasible on resource-constrained onboard hardware, whereas discarding it causes the target to drift or disappear. We address the tiny object and streaming challenges from both data and method perspectives. From the data perspective, we present \textbf{DroneEyes}, the \textbf{first} pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising $2,140$ high-definition videos and $176,623$ pairs across Object Description and Referring Expression tasks, with dense per-frame masks. From the method perspective, we propose \textbf{SkyAnchor}, an MLLM with two designs to the above challenges: a Semantics-Aware Token Router that preserves small-target under a reduced visual-token budget, and a Hierarchical Memory Bank that keeps the target consistently understood on streams.

Problem

• small object understanding
• streaming aerial videos
• memory augmentation
• multimodal large language models
• online perception

Innovation

• Memory-Augmented MLLM
• Tiny Object Understanding
• Streaming Aerial Video
• Semantics-Aware Token Router
• Hierarchical Memory Bank