In-place Double Stimulus Methodology for Subjective Assessment of High Quality Images

2025-08-13

AI Summary

Existing subjective evaluation methods struggle to reliably detect subtle perceptual differences in high-quality images. To address this, we propose the In-situ Dual-Stimulus (IDS) paradigm, which alternately presents reference and distorted images at the same spatial location to enhance human sensitivity to minute quality variations. We further introduce the IDS Quality Scale (IDSQS), the first subjective rating model to explicitly incorporate the Beta distribution—thereby jointly characterizing inter-observer consistency and intra-observer rating variability. Our approach integrates dual-stimulus temporal alternation, large-scale crowdsourced experiments, and principled statistical modeling, supported by an open-source GUI toolchain. Experiments demonstrate that IDSQS achieves exceptional correlation with fine-grained ground truth (Spearman’s ρ > 0.95), significantly improving fidelity-aware image quality assessment. Concurrently, we release the first high-quality public dataset specifically designed for evaluating subtle quality differences, alongside the complete open-source toolkit.

Technology Category

• Computer Vision: Image and Video Retrieval
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Search and Optimization: Distributed Search

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

This paper introduces a novel double stimulus subjective assessment methodology for the evaluation of high quality images to address the limitations of existing protocols in detecting subtle perceptual differences. The In-place Double Stimulus Quality Scale (IDSQS) allows subjects to alternately view a reference and a distorted image at the same spatial location, facilitating a more intuitive detection of differences in quality, especially at high to visually lossless quality levels. A large-scale crowdsourcing study employing this methodology was conducted, generating a comprehensive public dataset to evaluate perceived image quality across several compression algorithms and distortion levels. An additional contribution is the modeling of quality scores using a Beta distribution, allowing for the assessment of variability and subject consistency. Our findings demonstrate the effectiveness of the IDSQS methodology in achieving high correlation with more precise subjective evaluation benchmarks. The dataset, subjective data, and graphical user interface developed for this study are publicly available at https://github.com/shimamohammadi/IDSQS

Problem

• Develops a new method for assessing high-quality image differences
• Evaluates subtle perceptual differences in image compression algorithms
• Creates a public dataset for subjective image quality evaluation

Innovation

• In-place Double Stimulus Quality Scale methodology
• Large-scale crowdsourcing study for dataset
• Beta distribution models quality scores