ATLASv2: LLM-Guided Adaptive Landmark Acquisition and Navigation on the Edge

2025-04-15

AI Summary

To address the challenges of autonomous navigation and manipulation under severe resource constraints on edge devices, this paper proposes the first full-stack generative AI navigation system. The method integrates fine-tuned TinyLLM to build an online, self-growing landmark knowledge base; YOLOv5s for real-time object detection; a lightweight SLAM-inspired landmark localization module; and a hybrid A*/RRT path planner—enabling closed-loop coordination across perception, memory, planning, and execution. Its key contributions are (1) TinyLLM-driven dynamic landmark construction and natural-language task decomposition directly on-device, significantly enhancing environmental adaptability and task generalization; and (2) stable edge deployment in real-world home/office environments. Evaluated on Jetson Nano, the system achieves >92% task success rate, average prompt latency <320 ms, and power consumption <5.8 W—marking the first demonstration of generative AI navigation operating reliably at the edge.

Technology Category

• Planning, Routing, and Scheduling: Planning with Language Models
• Intelligent Robots: Localization, Mapping, and Navigation
• Machine Learning: Learning on the Edge & Model Compression

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Search and Retrieval-Augmented AI: Agentic search
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

Autonomous systems deployed on edge devices face significant challenges, including resource constraints, real-time processing demands, and adapting to dynamic environments. This work introduces ATLASv2, a novel system that integrates a fine-tuned TinyLLM, real-time object detection, and efficient path planning to enable hierarchical, multi-task navigation and manipulation all on the edge device, Jetson Nano. ATLASv2 dynamically expands its navigable landmarks by detecting and localizing objects in the environment which are saved to its internal knowledge base to be used for future task execution. We evaluate ATLASv2 in real-world environments, including a handcrafted home and office setting constructed with diverse objects and landmarks. Results show that ATLASv2 effectively interprets natural language instructions, decomposes them into low-level actions, and executes tasks with high success rates. By leveraging generative AI in a fully on-board framework, ATLASv2 achieves optimized resource utilization with minimal prompting latency and power consumption, bridging the gap between simulated environments and real-world applications.

Problem

• Enables autonomous navigation on resource-constrained edge devices
• Dynamically expands landmarks for adaptive task execution
• Integrates real-time object detection with efficient path planning

Innovation

• Integrates TinyLLM for edge AI tasks
• Uses real-time object detection for landmarks
• Implements efficient on-board path planning