Toward Semantic Communication for Real-time Mobile 3D Reconstruction

2026-07-17

AI Summary

This work addresses the degradation of multi-view geometric consistency and pose estimation accuracy in real-time mobile 3D reconstruction caused by communication distortions. To this end, we propose a semantic communication framework tailored for this task, wherein a joint semantic transceiver generates both reconstructed images and pixel-wise confidence maps, introducing pixel-level reliability information into semantic communication systems for the first time. Building upon this, we design a confidence-guided geometric estimation algorithm that integrates the confidence maps into RANSAC-based pose initialization and bundle adjustment to enhance 3D structural consistency. Experimental results demonstrate that, compared to existing semantic communication approaches and conventional separate encoding methods, our method significantly improves pose estimation accuracy and geometric robustness while maintaining high image fidelity.

Abstract

Real-time mobile 3D reconstruction is fundamental to many emerging applications such as autonomous navigation and digital twin construction, where a moving platform continuously captures an image stream and transmit to a computing server for scene understanding. Unlike offline reconstruction, camera poses and scene geometry are estimated on-the-fly during acquisition, making multi-view consistency a real-time requirement and rendering geometric estimation highly sensitive to communication-induced distortions. Semantic communication (SemCom) transmits compact semantic information, offering a promising way to preserve task-critical data over unreliable links. However, existing designs are optimized at the image or single-view level and without providing explicit reliability information for geometric estimation, limiting their applicability to real-time mobile 3D reconstruction. In this context, we propose a SemCom framework for real-time mobile 3D reconstruction. The framework includes a semantic transceiver that outputs a reconstructed image alongside a pixel-wise confidence map, quantifying the reliability of each region. We further introduce a confidence-guided geometric estimation method, incorporating confidence into RANSAC-based pose initialization and bundle adjustment to reduce the influence of unreliable regions and enhance robustness under noisy channels. Simulations show that, compared to existing SemCom and traditional seperate source and channel coding, our framework maintains high image quality while significantly improving pose estimation accuracy and 3D structural consistency.

Problem

• Semantic Communication
• Real-time 3D Reconstruction
• Mobile Systems
• Geometric Estimation
• Multi-view Consistency

Innovation

• Semantic Communication
• Real-time 3D Reconstruction
• Confidence Map
• Geometric Estimation
• Mobile Vision