LOCORE: Image Re-ranking with Long-Context Sequence Modeling

2025-03-27

AI Summary

This work addresses the image retrieval re-ranking task by proposing the first list-wise re-ranking method based on local descriptors. To overcome sequence-length limitations inherent in long-context modeling, we introduce a sliding-window inference strategy that jointly encodes local descriptors and employs an end-to-end learnable similarity scoring mechanism, enabling holistic modeling of the query image alongside the entire candidate list. Our approach achieves substantial performance gains over state-of-the-art re-rankers on standard benchmarks—including ROxf, RPar, SOP, In-Shop, and CUB-200—while maintaining inference latency comparable to mainstream pairwise local re-rankers. Key contributions include: (i) the first extension of list-wise re-ranking to the local descriptor level; (ii) a novel sliding-window attention mechanism tailored for long-context transformer-based models; and (iii) a unified design that simultaneously delivers high accuracy and computational efficiency.

Technology Category

• Computer Vision: Image and Video Retrieval
• Search and Optimization: Local Search
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs

Abstract

We introduce LOCORE, Long-Context Re-ranker, a model that takes as input local descriptors corresponding to an image query and a list of gallery images and outputs similarity scores between the query and each gallery image. This model is used for image retrieval, where typically a first ranking is performed with an efficient similarity measure, and then a shortlist of top-ranked images is re-ranked based on a more fine-grained similarity measure. Compared to existing methods that perform pair-wise similarity estimation with local descriptors or list-wise re-ranking with global descriptors, LOCORE is the first method to perform list-wise re-ranking with local descriptors. To achieve this, we leverage efficient long-context sequence models to effectively capture the dependencies between query and gallery images at the local-descriptor level. During testing, we process long shortlists with a sliding window strategy that is tailored to overcome the context size limitations of sequence models. Our approach achieves superior performance compared with other re-rankers on established image retrieval benchmarks of landmarks (ROxf and RPar), products (SOP), fashion items (In-Shop), and bird species (CUB-200) while having comparable latency to the pair-wise local descriptor re-rankers.

Problem

• Performs list-wise re-ranking with local descriptors for image retrieval
• Leverages long-context sequence models to capture query-gallery dependencies
• Achieves superior performance on multiple image retrieval benchmarks

Innovation

• List-wise re-ranking with local descriptors
• Long-context sequence modeling for dependencies
• Sliding window strategy for context limits