ColorSense: A Study on Color Vision in Machine Visual Recognition

2022-12-16

AI Summary

This work addresses the implicit reliance of machine vision on color in safety-critical applications (e.g., autonomous driving and surgical assistance), noting that existing studies are hindered by the absence of fine-grained color perception evaluation benchmarks. To bridge this gap, we introduce ColorSense—a large-scale, human-annotated dataset comprising 110,000 samples—and propose the first quantitative framework for color discrimination grading. We systematically evaluate mainstream vision models across classification and localization tasks to characterize color bias and robustness deficiencies. Leveraging multi-source data fusion, adversarial chromatic perturbation, and cross-architecture/task/scale attribution analysis, we find that color discrimination difficulty is a dominant factor governing model performance variance—especially degrading recognition of high-risk categories (e.g., vehicles)—while conventional data augmentation yields marginal improvements. Our findings call for a paradigm shift in color-robustness evaluation and provide theoretical foundations and standardized benchmarks for designing high-reliability vision systems.

Abstract

Color vision is essential for human visual perception, but its impact on machine perception is still underexplored. There has been an intensified demand for understanding its role in machine perception for safety-critical tasks such as assistive driving and surgery but lacking suitable datasets. To fill this gap, we curate multipurpose datasets ColorSense, by collecting 110,000 non-trivial human annotations of foreground and background color labels from popular visual recognition benchmarks. To investigate the impact of color vision on machine perception, we assign each image a color discrimination level based on its dominant foreground and background colors and use it to study the impact of color vision on machine perception. We validate the use of our datasets by demonstrating that the level of color discrimination has a dominating effect on the performance of mainstream machine perception models. Specifically, we examine the perception ability of machine vision by considering key factors such as model architecture, training objective, model size, training data, and task complexity. Furthermore, to investigate how color and environmental factors affect the robustness of visual recognition in machine perception, we integrate our ColorSense datasets with image corruptions and perform a more comprehensive visual perception evaluation. Our findings suggest that object recognition tasks such as classification and localization are susceptible to color vision bias, especially for high-stakes cases such as vehicle classes, and advanced mitigation techniques such as data augmentation and so on only give marginal improvement. Our analyses highlight the need for new approaches toward the performance evaluation of machine perception models in real-world applications. Lastly, we present various potential applications of ColorSense such as studying spurious correlations.

Problem

• Explores color vision's impact on machine perception.
• Addresses lack of datasets for color vision studies.
• Investigates color bias in critical recognition tasks.

Innovation

• Created ColorSense dataset
• Analyzed color discrimination impact
• Integrated datasets with image corruptions