MERGE: Multi-LLM Ensemble for Retrieval via Generative Enrichment

2026-09-29

AI Summary

This study addresses the limitations of single-LLM query augmentation, specifically architectural bias and poor prompt scalability, by proposing a two-stage multi-model ensemble framework. First, three heterogeneous open-source LLMs generate candidate query expansions, which are subsequently synthesized into a unified query by a larger model. Additionally, this work introduces task-oriented Automatic Prompt Optimization (APO) grounded in downstream retrieval performance, leveraging a tournament mechanism to enable scalable, human-free prompt engineering. Experimental results demonstrate that the proposed approach significantly improves BM25 nDCG@10 scores across five BEIR benchmarks, outperforming strong baselines. Ablation studies further validate the effectiveness of both the multi-model ensemble strategy and APO.

Abstract

Large Language Models (LLMs) are increasingly used to enrich user queries in information retrieval (IR) so that a standard retriever such as BM25 can bridge vocabulary gaps with the target corpus. Any single LLM, however, is limited by its training data and architectural biases, and its enrichment behavior depends on hand-crafted prompts that must be re-engineered for each new model -- an expensive and poorly scalable process. We present MERGE (Multi-LLM Ensemble for Retrieval via Generative Enrichment), a two-stage framework: three heterogeneous 7-8B open-source LLMs independently produce candidate expansions, and a larger LLM generatively synthesizes them into a single query. To make prompt engineering scalable across the ensemble, we integrate a task-grounded Automatic Prompt Optimization (APO) loop into both stages. Unlike APO methods that judge candidates with an LLM evaluator, our loop scores each candidate by its downstream retrieval performance and runs a small tournament between the current champion prompt and optimizer-proposed drafts, terminating once the champion survives two consecutive rounds; a history-augmented variant additionally feeds the recent tournament trajectory back to the optimizer. MERGE is retriever-agnostic and issues a single BM25 pass with no rank fusion, no supervised document expansion, and no re-indexing. On five BEIR benchmarks (NQ, SciFact, FiQA, Touche-2020, DBPedia), MERGE improves BM25 nDCG@10 over the original queries by +2.1 to +14.9 points and matches or outperforms strong LLM-based query-expansion baselines despite using only compact open-source models. Ablations confirm that the Stage-2 ensemble beats any single Stage-1 LLM, and that task-grounded APO converts large seed-prompt regressions into consistent gains without hand-tuning.

Problem

• Information Retrieval
• Query Expansion
• Large Language Models
• Prompt Engineering
• Vocabulary Gap

Innovation

• Multi-LLM Ensemble
• Generative Query Expansion
• Automatic Prompt Optimization
• Information Retrieval
• Tournament-based APO