Evaluating Facial Expression Recognition Datasets for Deep Learning: A Benchmark Study with Novel Similarity Metrics

2025-03-26

AI Summary

Limited data quality and diversity in facial expression recognition (FER) datasets critically constrain model generalizability. Method: We systematically evaluate the deep learning suitability of 24 mainstream FER datasets, introducing three novel similarity metrics—Local, Global, and Paired—to quantitatively characterize dataset difficulty, intra-dataset generalization, and cross-dataset transferability. A unified preprocessing pipeline, automated age/gender meta-annotation, and standardized deep neural network benchmarking ensure cross-dataset comparability. Contribution/Results: Experiments reveal that large-scale, in-the-wild datasets (e.g., AffectNet) yield superior generalization, whereas controlled-capture datasets—though precisely labeled—exhibit constrained representational diversity. This work establishes the first empirically grounded design guideline for FER dataset selection, construction, and benchmark evaluation.

Technology Category

Computer Vision: Biometrics, Face, Gesture & PoseNatural Language Processing: Interpretability, Analysis, and Evaluation of NLP ModelsMachine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasetsSearch and Retrieval-Augmented AI: Web evaluation methodologies and metricsWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

This study investigates the key characteristics and suitability of widely used Facial Expression Recognition (FER) datasets for training deep learning models. In the field of affective computing, FER is essential for interpreting human emotions, yet the performance of FER systems is highly contingent on the quality and diversity of the underlying datasets. To address this issue, we compiled and analyzed 24 FER datasets, including those targeting specific age groups such as children, adults, and the elderly, and processed them through a comprehensive normalization pipeline. In addition, we enriched the datasets with automatic annotations for age and gender, enabling a more nuanced evaluation of their demographic properties. To further assess dataset efficacy, we introduce three novel metricsLocal, Global, and Paired Similarity, which quantitatively measure dataset difficulty, generalization capability, and cross-dataset transferability. Benchmark experiments using state-of-the-art neural networks reveal that large-scale, automatically collected datasets (e.g., AffectNet, FER2013) tend to generalize better, despite issues with labeling noise and demographic biases, whereas controlled datasets offer higher annotation quality but limited variability. Our findings provide actionable recommendations for dataset selection and design, advancing the development of more robust, fair, and effective FER systems.

Problem

• Evaluating suitability of FER datasets for deep learning models
• Introducing novel metrics to measure dataset difficulty and generalization
• Providing recommendations for robust and fair FER system development

Innovation

• Comprehensive normalization pipeline for FER datasets
• Automatic annotations for age and gender
• Novel similarity metrics for dataset evaluation