NeRFCom: Feature Transform Coding Meets Neural Radiance Field for Free-View 3D Scene Semantic Transmission

2025-02-27

IEEE Communications Letters

AI Summary

Traditional NeRF-based semantic transmission for free-viewpoint 3D scenes suffers from bandwidth inefficiency and poor error resilience due to the decoupled design of NeRF compression and channel coding. To address this, we propose an end-to-end jointly optimized framework integrating neural radiance field (NeRF) semantic modeling, deep learning-driven nonlinear transform coding, and a learnable probabilistic entropy model—enabling semantic-aware, variable-bitrate joint source–channel coding that dynamically allocates bits according to semantic importance. This work is the first to co-model semantic-aware reconstruction quality and channel characteristics, ensuring high-fidelity free-viewpoint rendering even under adverse channel conditions. Experimental results demonstrate an average PSNR gain of 2.8 dB over conventional approaches, significantly improving both transmission efficiency and robustness.

Technology Category

Computer Vision: 3D Computer VisionCognitive Modeling & Cognitive Systems: Neural Spike CodingMachine Learning: Deep Generative Models & Autoencoders

Application Category

Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMsSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Graph embeddings and representation learning for Web-related graphs

Abstract

We introduce NeRFCom, a novel communication system designed for end-to-end 3D scene transmission. Compared to traditional systems relying on handcrafted NeRF semantic feature decomposition for compression and well-adaptive channel coding for transmission error correction, our NeRFCom employs a nonlinear transform and learned probabilistic models, enabling flexible variable-rate joint source-channel coding and efficient bandwidth allocation aligned with the NeRF semantic feature's different contribution to the 3D scene synthesis fidelity. Experimental results demonstrate that NeRFCom achieves free-view 3D scene efficient transmission while maintaining robustness under adverse channel conditions.

Problem

• End-to-end 3D scene transmission
• Flexible variable-rate joint source-channel coding
• Efficient bandwidth allocation for NeRF semantic features

Innovation

• Neural Radiance Field
• Nonlinear Transform Coding
• Joint Source-Channel Coding