LLM-VLM Fusion Framework for Autonomous Maritime Port Inspection using a Heterogeneous UAV-USV System

2026-01-19

AI Summary

This work proposes the first autonomous inspection framework that integrates large language models (LLMs) and vision-language models (VLMs) for heterogeneous drone–USV systems in port environments. Addressing the limitations of manual inspections and context-agnostic visual methods, the system interprets natural language instructions to generate symbolic task plans and performs real-time semantic compliance assessment with structured reporting. By replacing conventional finite-state machines with LLM-driven symbolic planning and VLM-based semantic understanding, the approach enables context-aware, adaptive, and resource-efficient collaborative inspection. Extensive validation on an extended MBZIRC maritime simulator and real robotic platforms demonstrates the system’s capability to safely and efficiently conduct real-time semantic inspections in complex port settings, highlighting its readiness for practical deployment.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Planning with Language Models
• Computer Vision: Language and Vision

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Maritime port inspection plays a critical role in ensuring safety, regulatory compliance, and operational efficiency in complex maritime environments. However, existing inspection methods often rely on manual operations and conventional computer vision techniques that lack scalability and contextual understanding. This study introduces a novel integrated engineering framework that utilizes the synergy between Large Language Models (LLMs) and Vision Language Models (VLMs) to enable autonomous maritime port inspection using cooperative aerial and surface robotic platforms. The proposed framework replaces traditional state-machine mission planners with LLM-driven symbolic planning and improved perception pipelines through VLM-based semantic inspection, enabling context-aware and adaptive monitoring. The LLM module translates natural language mission instructions into executable symbolic plans with dependency graphs that encode operational constraints and ensure safe UAV-USV coordination. Meanwhile, the VLM module performs real-time semantic inspection and compliance assessment, generating structured reports with contextual reasoning. The framework was validated using the extended MBZIRC Maritime Simulator with realistic port infrastructure and further assessed through real-world robotic inspection trials. The lightweight on-board design ensures suitability for resource-constrained maritime platforms, advancing the development of intelligent, autonomous inspection systems. Project resources (code and videos) can be found here: https://github.com/Muhayyuddin/llm-vlm-fusion-port-inspection

Problem

• maritime port inspection
• autonomous inspection
• contextual understanding
• scalability
• heterogeneous robotic systems

Innovation

• LLM-VLM fusion
• autonomous maritime inspection
• heterogeneous UAV-USV system
• symbolic planning
• semantic compliance assessment