Vision-Based Early Fault Diagnosis and Self-Recovery for Strawberry Harvesting Robots

2026-01-05

arXiv.org

AI Summary

This study addresses frequent harvesting failures in strawberry-picking robots caused by fragmented visual perception, misalignment between fruit and end-effector, empty grasps, and fruit slippage. To this end, the authors propose a fault diagnosis and self-recovery framework integrating multi-task visual perception with corrective control. They introduce SRR-Net, an end-to-end model that jointly performs strawberry detection, segmentation, and maturity estimation. Leveraging a micro-optical camera mounted on the gripper, the system incorporates a relative position error compensation mechanism between target and gripper, alongside a MobileNetV3-Small-based grasp state classifier and an LSTM-based slippage prediction module for early fault detection and real-time intervention. Experimental results demonstrate that SRR-Net achieves high performance in detection (precision: 0.895, recall: 0.813), segmentation (precision: 0.887, recall: 0.747), and maturity estimation (MAE: 0.035), with an inference speed of 163.35 FPS, significantly enhancing harvesting reliability and efficiency.

Technology Category

• Intelligent Robots: State Estimation
• Computer Vision: Vision for Robotics & Autonomous Driving
• Planning, Routing, and Scheduling: Replanning and Plan Repair

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Strawberry harvesting robots faced persistent challenges such as low integration of visual perception, fruit-gripper misalignment, empty grasping, and strawberry slippage from the gripper due to insufficient gripping force, all of which compromised harvesting stability and efficiency in orchard environments. To overcome these issues, this paper proposed a visual fault diagnosis and self-recovery framework that integrated multi-task perception with corrective control strategies. At the core of this framework was SRR-Net, an end-to-end multi-task perception model that simultaneously performed strawberry detection, segmentation, and ripeness estimation, thereby unifying visual perception with fault diagnosis. Based on this integrated perception, a relative error compensation method based on the simultaneous target-gripper detection was designed to address positional misalignment, correcting deviations when error exceeded the tolerance threshold. To mitigate empty grasping and fruit-slippage faults, an early abort strategy was implemented. A micro-optical camera embedded in the end-effector provided real-time visual feedback, enabling grasp detection during the deflating stage and strawberry slip prediction during snap-off through MobileNet V3-Small classifier and a time-series LSTM classifier. Experiments demonstrated that SRR-Net maintained high perception accuracy. For detection, it achieved a precision of 0.895 and recall of 0.813 on strawberries, and 0.972/0.958 on hands. In segmentation, it yielded a precision of 0.887 and recall of 0.747 for strawberries, and 0.974/0.947 for hands. For ripeness estimation, SRR-Net attained a mean absolute error of 0.035, while simultaneously supporting multi-task perception and sustaining a competitive inference speed of 163.35 FPS.

Problem

• strawberry harvesting robots
• visual perception
• fruit-gripper misalignment
• empty grasping
• fruit slippage

Innovation

• multi-task perception
• fault diagnosis and self-recovery
• SRR-Net
• relative error compensation
• grasp slip prediction