Retentive Relevance: Capturing Long-Term User Value in Recommendation Systems

2025-10-08

AI Summary

Traditional recommender systems rely on short-term interaction signals (e.g., clicks, likes), which suffer from high noise and sparsity, limiting their ability to model users’ long-term retention intent. To address this, we propose *Retentive Relevance*—the first content-level questionnaire-based feedback metric grounded in psychometric principles. It establishes, for the first time, a causal link between content perception and user revisit behavior, and rigorously validates its convergence, discriminability, and behavioral validity. We further design a lightweight surrogate model, integrated at the final stage of a multi-stage ranking pipeline, to calibrate scores and optimize online recommendations. A/B testing demonstrates statistically significant improvements: +3.2% in 7-day user retention, +4.1% in session engagement duration, and −12.7% reduction in low-quality content exposure. Notably, prediction accuracy for low-engagement users improves substantially (AUC +0.08).

Technology Category

Machine Learning: Learning Preferences or RankingsData Mining & Knowledge Management: Conversational Systems for Recommendation & RetrievalKnowledge Representation and Reasoning: Preferences

Application Category

User Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingWeb Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Recommendation systems have traditionally relied on short-term engagement signals, such as clicks and likes, to personalize content. However, these signals are often noisy, sparse, and insufficient for capturing long-term user satisfaction and retention. We introduce Retentive Relevance, a novel content-level survey-based feedback measure that directly assesses users' intent to return to the platform for similar content. Unlike other survey measures that focus on immediate satisfaction, Retentive Relevance targets forward-looking behavioral intentions, capturing longer term user intentions and providing a stronger predictor of retention. We validate Retentive Relevance using psychometric methods, establishing its convergent, discriminant, and behavioral validity. Through large-scale offline modeling, we show that Retentive Relevance significantly outperforms both engagement signals and other survey measures in predicting next-day retention, especially for users with limited historical engagement. We develop a production-ready proxy model that integrates Retentive Relevance into the final stage of a multi-stage ranking system on a social media platform. Calibrated score adjustments based on this model yield substantial improvements in engagement, and retention, while reducing exposure to low-quality content, as demonstrated by large-scale A/B experiments. This work provides the first empirically validated framework linking content-level user perceptions to retention outcomes in production systems. We offer a scalable, user-centered solution that advances both platform growth and user experience. Our work has broad implications for responsible AI development.

Problem

• Capturing long-term user value beyond short-term engagement signals
• Developing content-level feedback to predict user retention outcomes
• Integrating forward-looking behavioral intentions into recommendation systems

Innovation

• Introduces Retentive Relevance survey-based feedback measure
• Develops production-ready proxy model for ranking system
• Calibrates score adjustments to improve engagement and retention