A Comparative Evaluation of Embeddings and LLMs in a Greek Book Publisher Setting - The CUP Dataset

2026-07-23

AI Summary

This study addresses the lack of a realistic evaluation benchmark for Greek-language book retrieval by introducing CUP, the first dataset comprising 868 Greek bibliographic records and 104 expert-annotated queries. The authors systematically evaluate sparse (BM25), dense (sentence-transformers), hybrid, and large language model (LLM)-augmented retrieval approaches. Experimental results demonstrate that hybrid retrieval achieves the best overall performance; BM25 excels on named entity queries, while dense and hybrid methods substantially improve effectiveness on natural language, noisy, cross-lingual, and conceptual queries. Multilingual embeddings consistently outperform monolingual models, and LLM-based post-processing yields gains at a higher computational cost. This work establishes the first fine-grained retrieval benchmark for Greek publishing and provides a comprehensive comparison of modern retrieval strategies in this underexplored domain.

Abstract

We present CUP, a Greek book retrieval benchmark consisting of 868 catalog records and 104 expert-annotated queries with graded relevance judgments. We evaluate sparse (BM25), dense (sentence-transformers), hybrid, and LLM-assisted retrieval methods in this book-search setting. Multilingual embeddings outperform Greek-specific models, while hybrid retrieval performs best overall. A query-level analysis shows that BM25 excels at named-entity queries, while dense and hybrid methods improve natural-language, noisy, cross-lingual, and concept queries. Field-aware prompting has model-specific effects, while LLM TOC summarization improves TOC-only retrieval and LLM post-filtering improves early-stage retrieval at a high cost. Overall, CUP enables real-world evaluation of Greek retrieval across lexical, semantic, noisy, and cross-lingual queries.

Problem

• Greek book retrieval
• retrieval benchmark
• multilingual embeddings
• cross-lingual queries
• graded relevance judgments

Innovation

• Greek retrieval
• CUP dataset
• hybrid retrieval
• multilingual embeddings
• LLM-assisted retrieval