Find the Fruit: Designing a Zero-Shot Sim2Real Deep RL Planner for Occlusion Aware Plant Manipulation

2025-05-22

AI Summary

In dense agricultural environments with heavy occlusion of target objects (e.g., fruits) by foliage, conventional robotic harvesting systems struggle with reliable perception and manipulation due to limited visibility and complex plant dynamics. Method: This paper proposes an end-to-end deep reinforcement learning planner for zero-shot simulation-to-reality (sim2real) transfer. It introduces a novel decoupled architecture separating motion planning from low-level control, integrates vision with tactile/proprioceptive multimodal sensing, and enables occlusion-aware perception and active plant deformation—without requiring explicit plant geometry or dynamics modeling. A domain-adaptation-based transfer mechanism ensures fully zero-shot deployment. Contribution/Results: Evaluated in real orchards under diverse initial conditions, the system achieves an average success rate of 86.7%, significantly improving fruit localization and de-occlusion capability in cluttered vegetation. It establishes a generalizable, model-free, plug-and-play operational paradigm for agricultural robotics.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Planning, Routing, and Scheduling: Learning for Planning and Scheduling
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Experiences and lessons learnt from Web-based algorithms and system deployments
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work

Abstract

This paper presents an end-to-end deep reinforcement learning (RL) framework for occlusion-aware robotic manipulation in cluttered plant environments. Our approach enables a robot to interact with a deformable plant to reveal hidden objects of interest, such as fruits, using multimodal observations. We decouple the kinematic planning problem from robot control to simplify zero-shot sim2real transfer for the trained policy. Our results demonstrate that the trained policy, deployed using our framework, achieves up to 86.7% success in real-world trials across diverse initial conditions. Our findings pave the way toward autonomous, perception-driven agricultural robots that intelligently interact with complex foliage plants to"find the fruit"in challenging occluded scenarios, without the need for explicitly designed geometric and dynamic models of every plant scenario.

Problem

• Occlusion-aware robotic manipulation in cluttered plants
• Zero-shot sim2real transfer for deep RL policies
• Autonomous fruit detection in complex foliage environments

Innovation

• Zero-shot sim2real deep RL planner
• Multimodal observations for occlusion awareness
• Decoupled kinematic planning and robot control