🤖 AI Summary
User interaction frequency pruning—commonly applied to filter out low-activity users—introduces systematic biases in dataset characteristics and algorithm evaluation in recommender systems. Method: We conduct a comprehensive empirical study across five public datasets, each augmented with multiple levels of pruning. We train and evaluate 11 state-of-the-art recommendation algorithms under both standard offline evaluation and cross-pruning generalization settings. Contribution/Results: We quantitatively demonstrate that conventional pruning severely reduces user coverage, inflating model performance on pruned test sets while consistently degrading performance on unpruned, full-distribution test sets. The magnitude of evaluation distortion increases monotonically with pruning intensity. This work is the first to formally quantify this “evaluation distortion” phenomenon induced by data pruning. We advocate for cautious adoption of pruning heuristics and emphasize robust evaluation under the complete, unaltered data distribution. Our findings provide critical methodological guidance for establishing reliable experimental benchmarks in recommender system research.
📝 Abstract
Offline evaluations in recommender system research depend heavily on datasets, many of which are pruned, such as the widely used MovieLens collections. This thesis examines the impact of data pruning - specifically, removing users with fewer than a specified number of interactions - on both dataset characteristics and algorithm performance. Five benchmark datasets were analysed in both their unpruned form and at five successive pruning levels (5, 10, 20, 50, 100). For each coreset, we examined structural and distributional characteristics and trained and tested eleven representative algorithms. To further assess if pruned datasets lead to artificially inflated performance results, we also evaluated models trained on the pruned train sets but tested on unpruned data. Results show that commonly applied core pruning can be highly selective, leaving as little as 2% of the original users in some datasets. Traditional algorithms achieved higher nDCG@10 scores when both training and testing on pruned data; however, this advantage largely disappeared when evaluated on unpruned test sets. Across all algorithms, performance declined with increasing pruning levels when tested on unpruned data, highlighting the impact of dataset reduction on the performance of recommender algorithms.