NeRF in Robotics: A Survey

2024-05-02

arXiv.org

AI Summary

Neural Radiance Fields (NeRF) face critical challenges in robotics—including real-time inference, dynamic scene modeling, lightweight deployment, and tight integration with perception modules such as SLAM—hindering their practical adoption in embodied agents. Method: This work establishes the first dual-axis research framework for NeRF in robotics, structured along *application scenarios* and *technical evolution*, systematically identifying key gaps. We synthesize state-of-the-art variants—including dynamic, sparse, and incremental NeRFs—alongside neural rendering optimizations and SLAM-coordinated modeling techniques, emphasizing robustness and embedded-system feasibility. Contribution/Results: Through a comprehensive review of over 100 works, we distill core challenges: real-time rendering, motion-consistent reconstruction, and online scene updating. We propose concrete, deployable technical pathways bridging NeRF advancements with robotic system requirements, delivering both theoretical foundations and a systematic roadmap for NeRF-enabled embodied intelligence.

Technology Category

• Intelligent Robots: Embodied AI
• Computer Vision: Vision for Robotics & Autonomous Driving
• Natural Language Processing: Safety and Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Meticulous 3D environment representations have been a longstanding goal in computer vision and robotics fields. The recent emergence of neural implicit representations has introduced radical innovation to this field as implicit representations enable numerous capabilities. Among these, the Neural Radiance Field (NeRF) has sparked a trend because of the huge representational advantages, such as simplified mathematical models, compact environment storage, and continuous scene representations. Apart from computer vision, NeRF has also shown tremendous potential in the field of robotics. Thus, we create this survey to provide a comprehensive understanding of NeRF in the field of robotics. By exploring the advantages and limitations of NeRF, as well as its current applications and future potential, we hope to shed light on this promising area of research. Our survey is divided into two main sections: extit{The Application of NeRF in Robotics} and extit{The Advance of NeRF in Robotics}, from the perspective of how NeRF enters the field of robotics. In the first section, we introduce and analyze some works that have been or could be used in the field of robotics from the perception and interaction perspectives. In the second section, we show some works related to improving NeRF's own properties, which are essential for deploying NeRF in the field of robotics. In the discussion section of the review, we summarize the existing challenges and provide some valuable future research directions for reference.

Problem

• Exploring NeRFs' role in robotics for 3D environment representation
• Assessing NeRFs' advantages and limitations in robotic applications
• Identifying future research directions for NeRFs in robotics

Innovation

• Neural Radiance Fields for 3D representations
• Low memory footprint continuous scene modeling
• Enhanced robotics perception and interaction tasks