Transforming Video Subjective Testing with Training, Engagement, and Real-Time Feedback

2026-01-07

arXiv.org

AI Summary

This work proposes a three-stage subjective video quality assessment framework to address the limitations of traditional methods, which often fail to capture subtle perceptual differences and suffer from unreliable participant ratings. The framework integrates an automated training quiz, a real-time attention feedback mechanism based on “golden” video pairs, and an efficient chained pairwise comparison procedure, all grounded in the Just-Objectionable-Differences (JOD) scoring paradigm. Evaluated with 80 participants, the approach significantly improves accuracy in golden trial judgments, reduces tie rates, and mitigates non-monotonicity in the rate-quality (R-Q) curves within high-quality regions. These enhancements collectively yield higher-quality and more consistent subjective data, providing a robust foundation for training improved objective video quality metrics.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Machine Learning: Learning Preferences or Rankings

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

Subjective video quality assessment is crucial for optimizing streaming and compression, yet traditional protocols face limitations in capturing nuanced perceptual differences and ensuring reliable user input. We propose an integrated framework that enhances rater training, enforces attention through real-time scoring, and streamlines pairwise comparisons to recover quality scores with fewer comparisons. Participants first undergo an automated training quiz to learn key video quality indicators (e.g., compression artifacts) and verify their readiness. During the test, a real-time attention scoring mechanism, using"golden"video pairs, monitors and reinforces rater focus by applying penalties for lapses. An efficient chain-based pairwise comparison procedure is then employed, yielding quality scores in Just-Objectionable-Differences (JOD) units. Experiments comparing three groups (no training, training without feedback, and training with feedback) with 80 participants demonstrate that training-quiz significantly improves data quality in terms of golden unit accuracy and reduces tie rate, while real-time feedback further improves data quality and yields the most monotonic quality ratings. The new training, quiz, testing with feedback, 3-phase approach can significantly reduce the non-monotonic cases on the high quality part of the R-Q curve where normal viewer typically prefer the slightly compressed less-grainy content and help train a better objective video quality metric.

Problem

• subjective video quality assessment
• perceptual differences
• reliable user input
• non-monotonicity
• rate-quality curve

Innovation

• subjective video quality assessment
• real-time feedback
• rater training
• pairwise comparison
• Just-Objectionable-Differences (JOD)