offline recommendation metrics

Offline evaluation metrics in recommender systems often exhibit poor correlation with online performance, limiting their reliability for predicting real-world effectiveness. To address this misalignment, we propose a generic Pareto-frontier approximation strategy that jointly calibrates multiple offline metrics (e.g., Recall, NDCG) against multidimensional online metrics (e.g., CTR, CVR, GMV) within a single-model framework—without architectural modifications. Our method is model-agnostic, supports parallel A/B testing, and scales efficiently to industrial settings. Evaluated on large-scale production traffic at OTTO’s e-commerce platform, the approach significantly improves the consistency between offline metric trends and observed online outcomes. It provides an interpretable, reusable, and scientifically grounded foundation for metric selection and algorithmic iteration in industrial recommender systems.

Offline evaluation in recommender systems often fails to accurately reflect users’ true preferences due to data sparsity and various biases. This study systematically investigates how different offline evaluation designs—such as data filtering thresholds and candidate set construction strategies—affect model ranking outcomes. For the first time, it quantifies the convergent validity of these designs by measuring their ranking consistency against rankings derived from dense, real user feedback. Through extensive multi-configuration experiments and correlation analyses, the work demonstrates that the effectiveness of offline evaluation is highly dependent on the specific dataset and target task, with no universally optimal design. These findings underscore a critical principle: evaluation protocols must be carefully aligned with the intended application context.

Existing offline evaluation metrics for fairness in recommender systems lack systematic analysis of robustness, interpretability, and applicability, making it difficult to select and interpret them appropriately in practice. This work presents the first comprehensive examination of mainstream fairness metrics, uncovering their theoretical shortcomings and empirical limitations across user- and item-side perspectives as well as group- and individual-level granularities. By integrating multidimensional fairness definitions and employing theoretical derivations, distributional analyses, and boundary-case testing, the study delineates the operational boundaries and failure modes of these metrics. Building on these insights, the paper proposes refined evaluation methodologies and practical guidelines for metric selection, substantially enhancing the reliability and actionable utility of fairness assessments in recommender systems.

This work addresses the misalignment between offline evaluation metrics and online performance objectives in industrial applications by establishing a unified theoretical framework that systematically quantifies the relationships among diverse evaluation metrics for the first time. By introducing the concepts of Bayes-optimal sets and regret transfer mechanisms, the study reveals structural asymmetries among metrics and provides a principled classification and relational modeling of metrics with varying mathematical forms. Theoretically characterizing metric consistency and transferability, this research offers novel insights and a methodological foundation for designing offline evaluation systems that are aligned with online objectives and backed by rigorous theoretical guarantees.

In offline recommendation evaluation, exposure bias and sampling bias severely compromise assessment reliability, leading to distorted model performance comparisons. To address this, we propose a four-dimensional evaluation framework—discriminability, fidelity, robustness, and predictive power—to systematically quantify how sampling strategies distort evaluation outcomes. Leveraging fully observed ground-truth data, we simulate diverse exposure bias scenarios and empirically analyze the performance of mainstream sampling strategies—including uniform and popularity-based sampling. Experimental results reveal significant trade-offs across dimensions, with no universally optimal strategy. Accordingly, we provide a task-aware sampling strategy selection guideline tailored to specific evaluation objectives. This work establishes the first systematic, reliability-oriented evaluation paradigm for sampling strategies in offline recommendation, offering both theoretical foundations and practical guidance for trustworthy offline evaluation.

This study systematically investigates whether increasing training data scale consistently improves recommendation performance and whether a point of diminishing returns exists. Leveraging 11 large-scale public datasets, the authors evaluate 10 representative algorithms using LensKit and RecBole across varying sample sizes. To ensure rigor and reproducibility, they employ absolute stratified user sampling, min-max normalization, and late-stage slope analysis. Results show that approximately 75% of experimental configurations achieve peak performance at the largest data scales, with median late-stage slopes approaching 1.0. This work provides the first empirical evidence—on datasets comprising hundreds of millions of interactions—that traditional recommendation algorithms exhibit no clear saturation effect, underscoring the sustained positive impact of data scale on recommendation effectiveness.

This work addresses two key limitations in multi-domain algorithm performance evaluation: (i) the neglect of user preferences in assessment, and (ii) the masking of domain-specific performance disparities by conventional arithmetic averaging. To this end, we propose a user-preference-parameterized weighted scoring framework. Methodologically, we introduce, for the first time, a continuous family of scoring functions to model performance distributions; integrate probability measures with normalized confusion matrices; rigorously define four critical domain types—easiest, hardest, dominant, and bottleneck—and prove that only specific scoring functions preserve weighted mean consistency. Our contributions include: (i) establishing a general theoretical foundation for multi-domain performance analysis; (ii) developing a visualization toolkit tailored to binary classification tasks; and (iii) enabling fine-grained, interpretable performance decomposition—thereby substantially enhancing transparency and practical utility in cross-domain evaluation. (149 words)

Traditional Top-K offline recommendation evaluation relies on biased and incomplete user behavior ID matching, suffering from limited reliability and interpretability. This work proposes a novel evaluation framework grounded in large language models (LLMs): it constructs semantic preference proxies from users’ textual interactions, enables flexible matching of recommendations within a semantic space, and adopts a “reason-then-score” paradigm to generate relevance judgments accompanied by justifications. By replacing rigid ID-based matching with semantic alignment, the approach substantially enhances evaluation reliability while leveraging the LLM’s reasoning capabilities to yield interpretable scores. Experimental results demonstrate that the proposed framework consistently outperforms existing methods in terms of reliability, interpretability, and robustness, effectively identifying both successful and failed recommendations with well-reasoned explanations.

This study addresses the inconsistency in model rankings caused by commonly used ranking metrics—such as MRR, Hits@k, and Mean Rank—in knowledge graph completion (KGC) evaluation, which hinders fair comparison and reproducibility. For the first time, KGC evaluation is framed as a multi-criteria decision-making problem, and seven aggregators are systematically assessed across five dimensions: consistency, cross-dataset stability, metric independence, noise robustness, and generalization capability. Through leave-one-model-out (LOMO) and leave-one-group-out (LOGO) cross-validation, Pareto optimality analysis, and multidimensional sensitivity tests, the Z-score aggregator emerges as the most balanced overall—favoring DualE for tail entity prediction and FMS for relation prediction. The experiments further reveal that consistency and stability are insensitive to removal strategies, whereas generalization and independence exhibit the highest sensitivity.

Current evaluation of long-form question answering systems predominantly relies on human pairwise preference judgments, which often fail to capture the nuanced, expert-level assessment of in-depth research report quality. This work systematically examines the applicability and limitations of such meta-evaluation approaches in scientific QA using the ScholarQA-CS2 benchmark. The study finds that pairwise preferences are suitable only for system-level comparisons, whereas metric-level evaluation requires explicit dimension-wise annotations combined with domain-expert review. It identifies subjectivity as a central challenge and proposes a set of meta-evaluation design guidelines aligned with expert expectations, offering practical recommendations for future evaluation frameworks, annotator expertise matching, and reporting practices in deep research-oriented QA systems.