Robotic Strawberry Harvesting with Robust Vision and Deep Reinforcement Learning based Sim-to-Real Control

2026-05-22

AI Summary

This work addresses the challenges of severe occlusion, perceptual ambiguity, and reliance on conventional motion planners in strawberry harvesting within complex agricultural environments by proposing a closed-loop picking system that integrates robust vision with simulation-driven reinforcement learning. The authors introduce HRAttnEdge-YOLO26-seg, a novel instance segmentation model featuring a high-resolution P2 branch, segmentation-path attention, and edge-supervised prototypical learning, achieving a 10–14% performance gain on both self-collected and public datasets. Furthermore, they successfully deploy—for the first time—a goal-conditioned PPO policy trained in Isaac Lab directly onto a physical UR10e robot, enabling smooth joint-level command generation without any traditional motion planner. In greenhouse trials harvesting 281 strawberries, the system attained a 96.6% reach success rate, 91.3% grasp success rate, and an overall picking success rate of 84.3%.

Abstract

This study presents a closed-loop robotic strawberry harvesting system that combines a robust vision module, simulation-trained deep reinforcement learning (DRL) control, and ROS-based realrobot execution. For perception, we propose HRAttnEdge-YOLO26-seg, a modified YOLO26-seg architecture that incorporates a high-resolution P2 branch, segmentation-path attention, and edgesupervised prototype learning to improve instance segmentation in cluttered scenes. For control, we train a target-conditioned Proximal Policy Optimization (PPO) policy in Isaac Lab to produce smooth joint-position commands for a UR10e manipulator and deploy it on a UR10e robot for targetfruit reaching and harvesting. This simulation-based approach reduces hardware dependency, lowers development cost, and allows scalable policy training without exhaustive physical trials before real deployment. The proposed vision model demonstrated the highest overall performance among the evaluated methods. On both self-collected and public datasets, the model showed a 10 to 14% improvement in segmentation performance. In controlled in-house tests, the PPO controller produced stable and dynamically smoother motion than a inverse kinematics (IK)-based MoveIt baseline. In greenhouse trials, the proposed integrated system harvested 281 strawberries, achieving 96.6% reaching success, 91.3% grasp-and-pull success, and 84.3% overall harvesting success. These results illustrate that task-specific perception combined with simulation-trained PPO can serve as a practical and resource-efficient alternative to conventional planner-dependent reaching in manipulation, enabling reliable closed-loop robotic harvesting in complex agricultural environments.

Problem

• robotic harvesting
• strawberry picking
• instance segmentation
• sim-to-real
• agricultural robotics

Innovation

• deep reinforcement learning
• sim-to-real transfer
• instance segmentation
• robotic harvesting
• attention mechanism