Augmented Relevance Datasets with Fine-Tuned Small LLMs

2025-04-14

AI Summary

Manual annotation of query-document relevance is costly and poorly scalable. Method: This paper proposes an automatic relevance labeling and data augmentation framework based on lightweight fine-tuning of small-scale large language models (LLMs). It systematically employs parameter-efficient fine-tuning (PEFT) of compact LLMs for relevance modeling—replacing both manual annotation and expensive black-box models—while preserving annotation accuracy and substantially reducing computational and human costs. The framework comprises four components: LLM fine-tuning, fine-grained relevance assessment, synthetic data generation, and joint optimization with downstream ranking models. Results: The fine-tuned small LLM outperforms several proprietary large models in relevance judgment. When used to augment training data, it improves the downstream ranking model’s NDCG@10 by up to 12.7%, demonstrating the effectiveness and practicality of this efficient, high-quality data augmentation paradigm.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Building high-quality datasets and labeling query-document relevance are essential yet resource-intensive tasks, requiring detailed guidelines and substantial effort from human annotators. This paper explores the use of small, fine-tuned large language models (LLMs) to automate relevance assessment, with a focus on improving ranking models' performance by augmenting their training dataset. We fine-tuned small LLMs to enhance relevance assessments, thereby improving dataset creation quality for downstream ranking model training. Our experiments demonstrate that these fine-tuned small LLMs not only outperform certain closed source models on our dataset but also lead to substantial improvements in ranking model performance. These results highlight the potential of leveraging small LLMs for efficient and scalable dataset augmentation, providing a practical solution for search engine optimization.

Problem

• Automate query-document relevance assessment using small LLMs
• Improve ranking models via augmented training datasets
• Enhance dataset quality for search engine optimization

Innovation

• Fine-tuned small LLMs automate relevance assessment
• Enhanced datasets improve ranking model performance
• Outperforms closed-source models in experiments