Exploiting Radiance Fields for Grasp Generation on Novel Synthetic Views

2025-05-16

AI Summary

Single-view vision-based grasping suffers from inaccurate pose estimation and low coverage due to occlusions. Method: We propose a virtual multi-view augmentation method that requires no camera motion: a scene Gaussian Splatting radiance field is constructed from a single input image to synthesize geometrically consistent novel views in real time; these synthesized views are integrated into the GraspNet grasp representation and force-closure evaluation pipeline, replacing conventional physical multi-view acquisition. Contribution/Results: This work is the first to end-to-end integrate neural-rendered synthetic views into the grasp pose generation pipeline, significantly improving grasping robustness at zero hardware mobility cost. Evaluation on the GraspNet-1B dataset demonstrates that synthetic views uncover numerous previously undetected force-closure grasps, substantially increasing grasp coverage—validating the feasibility and effectiveness of single-image radiance field–assisted grasping.

Technology Category

• Computer Vision: Computational Photography, Image & Video Synthesis
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Multi-instance/Multi-view Learning

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Vision based robot manipulation uses cameras to capture one or more images of a scene containing the objects to be manipulated. Taking multiple images can help if any object is occluded from one viewpoint but more visible from another viewpoint. However, the camera has to be moved to a sequence of suitable positions for capturing multiple images, which requires time and may not always be possible, due to reachability constraints. So while additional images can produce more accurate grasp poses due to the extra information available, the time-cost goes up with the number of additional views sampled. Scene representations like Gaussian Splatting are capable of rendering accurate photorealistic virtual images from user-specified novel viewpoints. In this work, we show initial results which indicate that novel view synthesis can provide additional context in generating grasp poses. Our experiments on the Graspnet-1billion dataset show that novel views contributed force-closure grasps in addition to the force-closure grasps obtained from sparsely sampled real views while also improving grasp coverage. In the future we hope this work can be extended to improve grasp extraction from radiance fields constructed with a single input image, using for example diffusion models or generalizable radiance fields.

Problem

• Generating grasp poses from novel synthetic views using radiance fields
• Reducing time-cost by avoiding multiple physical camera movements
• Improving grasp coverage with additional context from synthesized views

Innovation

• Uses Radiance Fields for novel view synthesis
• Improves grasp poses with synthetic views
• Enhances grasp coverage and force-closure grasps