UniSHARP: Universal Sharp Monocular View Synthesis

2026-06-05

AI Summary

This work addresses the limitations of conventional monocular view synthesis methods, which are constrained by the pinhole camera model, by introducing a unified framework applicable to arbitrary imaging systems—including perspective, fisheye, and panoramic cameras. The approach operates within a consistent omnidirectional latent space, achieving continuous cross-camera view synthesis through joint implicit alignment of features and Gaussian primitives, coupled with a ray-based universal representation. Inspired by UniK3D, the model employs an encoder to extract 2D semantic and 3D geometric features, which are jointly decoded into a cloud of Gaussian primitives arranged along rays and radial distances. Evaluated on a new benchmark encompassing diverse imaging systems, the method significantly outperforms existing approaches, demonstrating strong effectiveness and generalization capability in generic monocular rendering tasks.

Abstract

In this work, we focus on extending SHARP, the popular photorealistic view synthesis method, for universal monocular rendering across a continuum of camera systems, from conventional perspective cameras to wide-field-of-view, fisheye and omnidirectional panoramic settings. To overcome the pinhole-specific assumptions of SHARP, our key idea is to align various images in a unified omnidirectional latent space. Thus, we propose UniSHARP, which performs implicit alignment in both feature and Gaussian spaces. Specifically, Gaussian primitives are arranged along rays and radial distances in a ray-based universal representation, while 2D semantic and 3D spatial features extracted from UniK3D-inspired encoders are jointly decoded to generate the complete Gaussian cloud. To comprehensively evaluate our method, we construct a benchmark covering diverse imaging systems across various scenes. The benchmark is further stratified by field of view (FoV) to enable fine-grained assessment of the universal monocular rendering task. Extensive experiments on the proposed benchmark demonstrate the effectiveness of UniSHARP, outperforming alternative methods by a large margin. The project page can be found at: https://insta360-research-team.github.io/Unisharp-website/

Problem

• monocular view synthesis
• universal rendering
• omnidirectional imaging
• camera systems
• field of view

Innovation

• universal monocular rendering
• omnidirectional latent space
• ray-based representation
• Gaussian splatting
• view synthesis