Uncertainty as Remedy: Mitigating Satisfaction Label Bias in Short Video Multi-Objective Ensemble Ranking

2026-07-19

AI Summary

This work addresses the challenge of label bias in short-video recommendation, where multi-objective behavioral signals—often used as proxies for user satisfaction—suffer from partiality, fragmentation, and internal conflicts, thereby hindering model convergence. To mitigate this, the authors propose the UAME framework, which uniquely integrates prediction uncertainty directly into the core optimization process. By modeling user responses with Gaussian distributions—where the mean represents estimated satisfaction and the variance captures uncertainty—the framework introduces a probabilistic pairwise ranking loss coupled with an uncertainty-aware sample weighting mechanism, enabling end-to-end multi-objective ensemble ranking. Evaluated on a large-scale industrial platform, UAME significantly improves EMER and EASQ metrics, demonstrates stronger alignment with survey-based satisfaction scores, and has been deployed in production, delivering consistent performance gains.

Abstract

The core objective of short video recommendation is to model users' unobservable true satisfaction with recommended videos. As the dominant industrial framework, end-to-end multi-objective ensemble ranking models are typically trained with multi-dimensional dense user behavioral signals, such as clicks and watch time. However, these behavioral signals are partial, fragmented, and often mutually conflicting user satisfaction proxies, introducing uncertainty and label bias into satisfaction modeling. Conventional deterministic models overlook this uncertainty, which exacerbates satisfaction label bias and results in suboptimal model convergence. Meanwhile, existing uncertainty-aware methods mostly employ uncertainty for post-hoc ranking adjustments rather than leveraging it as a remedy to mitigate the inherent bias within the core optimization pipeline. This paper proposes UAME, an Uncertainty-Aware end-to-end Multi-objective Ensemble ranking framework for short video recommendation. UAME represents the model's prediction as a Gaussian scoring variable, where the mean denotes the predicted satisfaction score and the variance quantifies predictive uncertainty associated with this score. We further design a probabilistic pairwise ranking loss, and construct an uncertainty-aware sample-level weighting scheme to mitigate the bias. We further provide theoretical analysis suggesting that the weighting scheme helps mitigate satisfaction label bias. Extensive offline and online experiments on a large-scale industrial short video platform demonstrate that UAME consistently improves two state-of-the-art paradigms, EMER and EASQ, and better aligns with questionnaire-based user satisfaction. UAME has been deployed in our production short-video recommendation system and continues to deliver stable, statistically significant gains.

Problem

• satisfaction label bias
• uncertainty
• short video recommendation
• multi-objective ensemble ranking
• user behavioral signals

Innovation

• Uncertainty-aware learning
• Label bias mitigation
• Multi-objective ensemble ranking
• Probabilistic pairwise loss
• Short video recommendation