🤖 AI Summary
This study systematically investigates whether increasing training data scale consistently improves recommendation performance and whether a point of diminishing returns exists. Leveraging 11 large-scale public datasets, the authors evaluate 10 representative algorithms using LensKit and RecBole across varying sample sizes. To ensure rigor and reproducibility, they employ absolute stratified user sampling, min-max normalization, and late-stage slope analysis. Results show that approximately 75% of experimental configurations achieve peak performance at the largest data scales, with median late-stage slopes approaching 1.0. This work provides the first empirical evidence—on datasets comprising hundreds of millions of interactions—that traditional recommendation algorithms exhibit no clear saturation effect, underscoring the sustained positive impact of data scale on recommendation effectiveness.
📝 Abstract
In recommender systems, collecting, storing, and processing large-scale interaction data is increasingly costly in terms of time, energy, and computation, yet it remains unclear when additional data stops providing meaningful gains. This paper investigates how offline recommendation performance evolves as the size of the training dataset increases and whether a saturation point can be observed. We implemented a reproducible Python evaluation workflow with two established toolkits, LensKit and RecBole, included 11 large public datasets with at least 7 million interactions, and evaluated 10 tool-algorithm combinations. Using absolute stratified user sampling, we trained models on nine sample sizes from 100,000 to 100,000,000 interactions and measured NDCG@10. Overall, raw NDCG usually increased with sample size, with no observable saturation point. To make result groups comparable, we applied min-max normalization within each group, revealing a clear positive trend in which around 75% of the points at the largest completed sample size also achieved the group's best observed performance. A late-stage slope analysis over the final 10-30% of each group further supported this upward trend: the interquartile range remained entirely non-negative with a median near 1.0. In summary, for traditional recommender systems on typical user-item interaction data, incorporating more training data remains primarily beneficial, while weaker scaling behavior is concentrated in atypical dataset cases and in the algorithmic outlier RecBole BPR under our setup.