PairingNet: A Learning-based Pair-searching and -matching Network for Image Fragments

2023-12-14

arXiv.org

AI Summary

This paper addresses the challenging problem of automatic image fragment pairing and stitching for image restoration. We propose the first end-to-end jointly optimized framework that simultaneously solves fragment-pair search and geometric matching. Methodologically, we innovatively integrate a Graph Neural Network (GNN) with a linear Transformer to jointly model contour and texture features; we further introduce a dynamically weighted feature fusion module and a contrastive learning–driven global encoder, eliminating reliance on handcrafted rules and manual hyperparameter tuning. Experiments on our newly constructed irregular-fragment dataset demonstrate significant improvements: higher pairing accuracy, reduced geometric matching error, and substantially accelerated inference speed. Our core contributions include (1) unified modeling of pairing and alignment within a single architecture, (2) a learnable, adaptive feature fusion mechanism, and (3) an efficient, lightweight joint optimization paradigm that advances both accuracy and computational efficiency in fragment reconstruction.

Abstract

In this paper, we propose a learning-based image fragment pair-searching and -matching approach to solve the challenging restoration problem. Existing works use rule-based methods to match similar contour shapes or textures, which are always difficult to tune hyperparameters for extensive data and computationally time-consuming. Therefore, we propose a neural network that can effectively utilize neighbor textures with contour shape information to fundamentally improve performance. First, we employ a graph-based network to extract the local contour and texture features of fragments. Then, for the pair-searching task, we adopt a linear transformer-based module to integrate these local features and use contrastive loss to encode the global features of each fragment. For the pair-matching task, we design a weighted fusion module to dynamically fuse extracted local contour and texture features, and formulate a similarity matrix for each pair of fragments to calculate the matching score and infer the adjacent segment of contours. To faithfully evaluate our proposed network, we created a new image fragment dataset through an algorithm we designed that tears complete images into irregular fragments. The experimental results show that our proposed network achieves excellent pair-searching accuracy, reduces matching errors, and significantly reduces computational time. Details, sourcecode, and data are available in our supplementary material.

Problem

• Image Restoration
• Accuracy
• Efficiency

Innovation

• PairingNet
• Feature Extraction
• Contrastive Loss