Visual Language Models as Operator Agents in the Space Domain

2025-01-14

AI Summary

This work addresses the lack of cross-modal semantic understanding in autonomous decision-making and control for space missions. We propose the first framework systematically integrating vision-language models (VLMs) into a closed-loop space operations architecture. Methodologically, we unify multimodal prompt engineering, the Kerbal Space Program differential game (KSPDG) simulation environment, and an embedded robotic vision system to establish an end-to-end “perception–understanding–decision–execution” pipeline. Our approach overcomes the dual limitations of large language models (LLMs)—lacking visual grounding—and purely vision-based methods—lacking semantic reasoning. Experiments demonstrate that VLM-based agents achieve orbital maneuvering performance comparable to domain-specific algorithms and non-multimodal LLMs in KSPDG. Furthermore, on a physical robotic platform, the system successfully identifies satellite components and localizes anomalies, constituting the first experimental validation of VLM-driven autonomous on-orbit inspection and fault diagnosis.

Technology Category

• Computer Vision: Language and Vision
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

This paper explores the application of Vision-Language Models (VLMs) as operator agents in the space domain, focusing on both software and hardware operational paradigms. Building on advances in Large Language Models (LLMs) and their multimodal extensions, we investigate how VLMs can enhance autonomous control and decision-making in space missions. In the software context, we employ VLMs within the Kerbal Space Program Differential Games (KSPDG) simulation environment, enabling the agent to interpret visual screenshots of the graphical user interface to perform complex orbital maneuvers. In the hardware context, we integrate VLMs with robotic systems equipped with cameras to inspect and diagnose physical space objects, such as satellites. Our results demonstrate that VLMs can effectively process visual and textual data to generate contextually appropriate actions, competing with traditional methods and non-multimodal LLMs in simulation tasks, and showing promise in real-world applications.

Problem

• Visual Language Models
• Space Missions
• Decision and Control

Innovation

• Visual Language Models
• Space Missions
• Image Understanding