Hierarchical Tri-manual Planning for Vision-assisted Fruit Harvesting with Quadrupedal Robots

2024-09-25

arXiv.org

AI Summary

To address the low harvesting efficiency and poor environmental adaptability of dual-arm robots in natural orchards, this paper introduces LocoHarv-3—the first tri-arm, quadrupedal harvesting robot. Methodologically, we propose a hierarchical tri-arm coordinated motion planning framework integrating LiDAR-based mapping and odometry localization, learning-based visual perception (including fruit detection and 6D pose estimation), and semi-autonomous multimodal teleoperation to enable collision-free, vision-guided fully autonomous harvesting. The core contribution lies in overcoming the inherent limitations of dual-arm configurations through temporal task division and spatial coordination among three arms, significantly enhancing operational robustness and workspace reachability. Experimental results demonstrate a 90% single-fruit harvesting success rate in controlled laboratory settings. Field trials in unstructured real-world orchards further validate the system’s stability and efficiency under challenging conditions—including uneven terrain, occlusions from foliage, and dynamic environmental variability.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Vision for Robotics & Autonomous Driving
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling

Application Category

• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

This paper addresses the challenge of developing a multi-arm quadrupedal robot capable of efficiently harvesting fruit in complex, natural environments. To overcome the inherent limitations of traditional bimanual manipulation, we introduce the first three-arm quadrupedal robot LocoHarv-3 and propose a novel hierarchical tri-manual planning approach, enabling automated fruit harvesting with collision-free trajectories. Our comprehensive semi-autonomous framework integrates teleoperation, supported by LiDAR-based odometry and mapping, with learning-based visual perception for accurate fruit detection and pose estimation. Validation is conducted through a series of controlled indoor experiments using motion capture and extensive field tests in natural settings. Results demonstrate a 90% success rate in in-lab settings with a single attempt, and field trials further verify the system's robustness and efficiency in more challenging real-world environments.

Problem

• Developing multi-arm quadrupedal robot for fruit harvesting
• Overcoming limitations of traditional bimanual manipulation
• Integrating teleoperation with learning-based visual perception

Innovation

• Three-arm quadrupedal robot for fruit harvesting
• Hierarchical tri-manual planning for collision-free trajectories
• LiDAR and learning-based vision for fruit detection