🤖 AI Summary
Evaluating 3D reconstruction quality without ground-truth images faces two key challenges: the absence of reliable reference views and the inability of prevailing no-reference metrics to localize fine-grained artifacts in novel synthesized views. To address this, we propose Puzzle Similarity (PS), the first no-reference metric capable of spatially localizing artifact regions. Its core innovation lies in modeling scene-specific distributions via local patch statistics extracted from the input multi-view images, enabling adaptive distribution matching for fine-grained, reference-free artifact localization. Evaluated on human-perception datasets, PS significantly outperforms existing full-reference and no-reference metrics, achieving high correlation with subjective quality judgments. Crucially, PS operates entirely without ground-truth imagery, making it suitable for downstream applications such as artifact-driven automatic restoration, acquisition optimization, and sparse-view reconstruction.
📝 Abstract
Modern reconstruction techniques can effectively model complex 3D scenes from sparse 2D views. However, automatically assessing the quality of novel views and identifying artifacts is challenging due to the lack of ground truth images and the limitations of no-reference image metrics in predicting detailed artifact maps. The absence of such quality metrics hinders accurate predictions of the quality of generated views and limits the adoption of post-processing techniques, such as inpainting, to enhance reconstruction quality. In this work, we propose a new no-reference metric, Puzzle Similarity, which is designed to localize artifacts in novel views. Our approach utilizes image patch statistics from the input views to establish a scene-specific distribution that is later used to identify poorly reconstructed regions in the novel views. We test and evaluate our method in the context of 3D reconstruction; to this end, we collected a novel dataset of human quality assessment in unseen reconstructed views. Through this dataset, we demonstrate that our method can not only successfully localize artifacts in novel views, correlating with human assessment, but do so without direct references. Surprisingly, our metric outperforms both no-reference metrics and popular full-reference image metrics. We can leverage our new metric to enhance applications like automatic image restoration, guided acquisition, or 3D reconstruction from sparse inputs.