🤖 AI Summary
To address domain shift between synthetic and real-world images and high annotation costs in robotic vision tasks, this paper proposes an automated training dataset generation pipeline tailored for robotic environments. Methodologically, it introduces a novel two-pass rendering framework based on 3D Gaussian Splatting, integrating proxy-mesh shadow mapping with splatting-based image synthesis to achieve physically plausible shadow and highlight modeling. Concurrently, it generates pixel-accurate segmentation masks compatible with mainstream detectors such as YOLO. By training on a hybrid dataset comprising a small set of real images and large-scale, high-fidelity synthetic data, the pipeline significantly improves object detection and instance segmentation accuracy. Experiments demonstrate that the approach effectively bridges the domain gap while maintaining high rendering efficiency, offering a scalable, efficient paradigm for building robust robotic vision models.
📝 Abstract
This paper introduces a novel pipeline for generating large-scale, highly realistic, and automatically labeled datasets for computer vision tasks in robotic environments. Our approach addresses the critical challenges of the domain gap between synthetic and real-world imagery and the time-consuming bottleneck of manual annotation. We leverage 3D Gaussian Splatting (3DGS) to create photorealistic representations of the operational environment and objects. These assets are then used in a game engine where physics simulations create natural arrangements. A novel, two-pass rendering technique combines the realism of splats with a shadow map generated from proxy meshes. This map is then algorithmically composited with the image to add both physically plausible shadows and subtle highlights, significantly enhancing realism. Pixel-perfect segmentation masks are generated automatically and formatted for direct use with object detection models like YOLO. Our experiments show that a hybrid training strategy, combining a small set of real images with a large volume of our synthetic data, yields the best detection and segmentation performance, confirming this as an optimal strategy for efficiently achieving robust and accurate models.