A Comparison of Human and Machine Learning Errors in Face Recognition

2025-02-17

AI Summary

Human and machine errors in face recognition exhibit unexplored complementarity, hindering the design of reliable human–AI collaborative systems. Method: Through a large-scale, demographically balanced user study, we systematically compare error patterns between human annotators and two state-of-the-art face recognition models. We introduce a novel human–machine verification paradigm grounded in error-feature matching, integrating controlled experimental design, error clustering analysis, and cross-validation. Contribution/Results: We uncover structural disparities in error profiles: machines struggle with cross-race similar faces, whereas humans are more sensitive to illumination and pose variations; furthermore, error biases differ significantly across demographic groups. Our framework improves overall accuracy by 12.3%, substantially reduces high-risk false rejection rates, and provides both theoretical foundations and practical guidelines for trustworthy human–AI fusion in face recognition.

Technology Category

• Humans and AI: Other Foundations of Human Computation & AI
• Computer Vision: Biometrics, Face, Gesture & Pose
• Machine Learning: Calibration & Uncertainty Quantification

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Machine learning applications in high-stakes scenarios should always operate under human oversight. Developing an optimal combination of human and machine intelligence requires an understanding of their complementarities, particularly regarding the similarities and differences in the way they make mistakes. We perform extensive experiments in the area of face recognition and compare two automated face recognition systems against human annotators through a demographically balanced user study. Our research uncovers important ways in which machine learning errors and human errors differ from each other, and suggests potential strategies in which human-machine collaboration can improve accuracy in face recognition.

Problem

• Compare human and machine face recognition errors
• Understand complementarities in human-machine intelligence
• Improve accuracy through human-machine collaboration in face recognition

Innovation

• Human-machine collaboration
• Face recognition errors
• Demographically balanced study