🤖 AI Summary
Offline evaluation metrics in recommender systems often exhibit poor correlation with online performance, limiting their reliability for predicting real-world effectiveness. To address this misalignment, we propose a generic Pareto-frontier approximation strategy that jointly calibrates multiple offline metrics (e.g., Recall, NDCG) against multidimensional online metrics (e.g., CTR, CVR, GMV) within a single-model framework—without architectural modifications. Our method is model-agnostic, supports parallel A/B testing, and scales efficiently to industrial settings. Evaluated on large-scale production traffic at OTTO’s e-commerce platform, the approach significantly improves the consistency between offline metric trends and observed online outcomes. It provides an interpretable, reusable, and scientifically grounded foundation for metric selection and algorithmic iteration in industrial recommender systems.
📝 Abstract
A critical challenge in recommender systems is to establish reliable relationships between offline and online metrics that predict real-world performance. Motivated by recent advances in Pareto front approximation, we introduce a pragmatic strategy for identifying offline metrics that align with online impact. A key advantage of this approach is its ability to simultaneously serve multiple test groups, each with distinct offline performance metrics, in an online experiment controlled by a single model. The method is model-agnostic for systems with a neural network backbone, enabling broad applicability across architectures and domains. We validate the strategy through a large-scale online experiment in the field of session-based recommender systems on the OTTO e-commerce platform. The online experiment identifies significant alignments between offline metrics and real-word click-through rate, post-click conversion rate and units sold. Our strategy provides industry practitioners with a valuable tool for understanding offline-to-online metric relationships and making informed, data-driven decisions.