On the Convergent Validity of Offline Evaluation Designs for Recommender Systems

2026-07-27

AI Summary

Offline evaluation in recommender systems often fails to accurately reflect users’ true preferences due to data sparsity and various biases. This study systematically investigates how different offline evaluation designs—such as data filtering thresholds and candidate set construction strategies—affect model ranking outcomes. For the first time, it quantifies the convergent validity of these designs by measuring their ranking consistency against rankings derived from dense, real user feedback. Through extensive multi-configuration experiments and correlation analyses, the work demonstrates that the effectiveness of offline evaluation is highly dependent on the specific dataset and target task, with no universally optimal design. These findings underscore a critical principle: evaluation protocols must be carefully aligned with the intended application context.

Abstract

Offline evaluation on historical interaction logs is the most common evaluation methodology for recommender systems. However, such evaluations depend on sparse, incomplete, or biased data, which raises concerns about whether commonly used evaluation setups reliably reflect true user preferences. In this work, we study how offline evaluation design choices affect the validity of recommender system comparisons. We evaluate a set of recommendation models across several evaluation setups that vary key factors such as data filtering thresholds and candidate set construction. To assess the validity of these configurations, we measure the correlation between model rankings obtained from conventional train-test splits on sparse interaction data and rankings from evaluations based on dense ground-truth user feedback. We use this agreement as an indication of their validity with respect to true user preferences. Our results show that the validity of sparse evaluation depends on the dataset and the specific dense evaluation targets, and that there is no uniformly best offline evaluation design.

Problem

• offline evaluation
• recommender systems
• convergent validity
• user preferences
• evaluation design

Innovation

• offline evaluation
• recommender systems
• convergent validity
• evaluation design
• user feedback