Rank1: Test-Time Compute for Reranking in Information Retrieval

2025-02-25

AI Summary

This work addresses three key challenges in information retrieval reranking: weak reasoning capability, poor interpretability, and insufficient out-of-distribution (OOD) generalization. To this end, we propose Rank1—the first lightweight reranker incorporating test-time computation. Methodologically, we distill structured reasoning traces (>600K samples) from reasoning-oriented large language models (e.g., o1, R1), perform supervised training on MS MARCO, and support prompt-driven zero-shot transfer. The model architecture is designed for promptability, explicit interpretability, and inference efficiency; quantization further reduces computational and memory overhead. Key contributions include: (1) the first application of the test-time computation paradigm to reranking; (2) generation of human-readable, step-by-step reasoning chains as explicit outputs; and (3) state-of-the-art performance across multiple benchmarks with strong OOD generalization.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

We introduce Rank1, the first reranking model trained to take advantage of test-time compute. Rank1 demonstrates the applicability within retrieval of using a reasoning language model (i.e. OpenAI's o1, Deepseek's R1, etc.) for distillation in order to rapidly improve the performance of a smaller model. We gather and open-source a dataset of more than 600,000 examples of R1 reasoning traces from queries and passages in MS MARCO. Models trained on this dataset show: (1) state-of-the-art performance on advanced reasoning and instruction following datasets; (2) work remarkably well out of distribution due to the ability to respond to user-input prompts; and (3) have explainable reasoning chains that can be given to users or RAG-based systems. Further, we demonstrate that quantized versions of these models retain strong performance while using less compute/memory. Overall, Rank1 shows that test-time compute allows for a fundamentally new type of explainable and performant reranker model for search.

Problem

• Enhancing reranking performance using test-time compute.
• Improving small model efficiency via reasoning model distillation.
• Developing explainable reranking models for search applications.

Innovation

• Test-time compute reranking model
• Reasoning language model distillation
• Quantized model performance retention