🤖 AI Summary
To address the limited robustness of object instance segmentation in robotic grasping under dynamic scenes—where occlusion, low illumination, motion blur, and scale variation severely degrade performance—this paper proposes a dual-modal instance segmentation method fusing event camera data and RGB frames. We design a novel two-stream encoder architecture that jointly integrates spatial pyramid pooling (SPP) and dilated convolutions to enable cross-modal feature complementarity and multi-scale contextual modeling. Evaluated on the ESD dataset across five degradation scenarios, our method achieves 6–10% improvements in mean IoU and pixel accuracy over state-of-the-art approaches, significantly enhancing segmentation reliability in complex dynamic environments. To the best of our knowledge, this is the first work to introduce the synergistic integration of event-RGB dual-modal encoding with SPP-dilated convolution mechanisms into grasp-oriented instance segmentation, establishing a new paradigm for real-time, robust visual perception in robotics.
📝 Abstract
Object segmentation for robotic grasping under dynamic conditions often faces challenges such as occlusion, low light conditions, motion blur and object size variance. To address these challenges, we propose a Deep Learning network that fuses two types of visual signals, event-based data and RGB frame data. The proposed Bimodal SegNet network has two distinct encoders, one for each signal input and a spatial pyramidal pooling with atrous convolutions. Encoders capture rich contextual information by pooling the concatenated features at different resolutions while the decoder obtains sharp object boundaries. The evaluation of the proposed method undertakes five unique image degradation challenges including occlusion, blur, brightness, trajectory and scale variance on the Event-based Segmentation (ESD) Dataset. The evaluation results show a 6-10% segmentation accuracy improvement over state-of-the-art methods in terms of mean intersection over the union and pixel accuracy. The model code is available at https://github.com/sanket0707/Bimodal-SegNet.git
