face recognition benchmarking

Existing robustness evaluation methods for face recognition rely heavily on domain expertise, are time-consuming, and suffer from strong system coupling. Method: This paper proposes RobFace—the first system-agnostic, lightweight test suite for face recognition robustness assessment. Its core innovations include: (1) a transferable adversarial face image generation framework enabling cross-model generalization evaluation in black-box settings; (2) a multi-granularity evaluation framework covering perturbation types, intensities, and semantic dimensions; and (3) integration of empirical testing with formal analysis to ensure assessment consistency. Results: Experiments across multiple mainstream face recognition systems demonstrate that RobFace’s evaluation scores correlate strongly with actual attack success rates (Pearson > 0.92), while reducing evaluation time by over 90% compared to state-of-the-art tools. RobFace thus establishes a practical, third-party robustness benchmark—filling a critical gap in the field.

This study investigates how preprocessing components—particularly face detection models and image interpolation strategies—impede the transferability of adversarial attacks in black-box face recognition systems. We systematically evaluate the degradation in transfer success rates of mainstream white-box attack methods under black-box conditions, observing up to a 78% drop. To address this, we propose a preprocessing-invariant adversarial attack framework that employs differentiable input transformations to enhance the robustness of adversarial examples against diverse preprocessing pipelines. Experiments demonstrate that our method improves average attack transfer success by up to 27% across cross-detector and cross-interpolation scenarios. This work is the first to identify and characterize the critical role of the preprocessing chain in the white-box-to-black-box attack performance decay. It further establishes a novel, more robust paradigm for adversarial attack and defense evaluation in real-world face recognition systems.

This study addresses the privacy and ethical challenges inherent in facial recognition evaluation by systematically assessing the viability of synthetic face datasets as alternatives to real human data. Evaluating twelve synthetic datasets across verification metrics, similarity distributions, model ranking consistency, and data distribution characteristics, the work conducts a comprehensive comparison using twenty-four pre-trained models—spanning both CNN and Transformer architectures—on seven real and twelve synthetic benchmarks. The findings demonstrate for the first time that high-quality synthetic datasets such as MorphFace and Vec2Face achieve evaluation reliability comparable to real-world benchmarks, with model behavior reproducibility matching the natural variation observed between real datasets. These results establish a feasible pathway toward fully privacy-preserving training and evaluation pipelines in facial recognition research.

Facing the bottleneck of facial morphing attack detection (MAD) development—namely, scarcity of authentic data and stringent privacy constraints—this work introduces the first large-scale, publicly available synthetic face morphing dataset, comprising 2,450 identities and over 100,000 high-fidelity samples, covering both single-sample and differential morphing attack scenarios. We propose a novel morphing synthesis framework grounded in generative modeling and deformation mapping, enabling, for the first time, systematic generation of mated-sample morphs. Furthermore, we unify image quality assessment (IQA) and vulnerability analysis to jointly optimize performance across both MAD tasks. Extensive multi-protocol benchmarking demonstrates that our dataset significantly outperforms existing state-of-the-art synthetic datasets, achieving detection accuracy comparable to that attained on real-world data—thereby validating its effectiveness and strong generalizability for training diverse MAD models.

Existing mainstream face verification benchmarks (e.g., LFW) suffer from identity/image overlap with training data, leading to inflated performance estimates and poor reflection of real-world challenges—such as large intra-class attribute variation, high inter-class visual similarity (e.g., relatives/twins), and fine-grained discrimination difficulty. Method: This work systematically characterizes the verification bias induced by non-isolated identities and introduces two rigorously constructed “Goldilocks” benchmarks—Hadrian (controlling hairstyle variation) and Eclipse (controlling illumination/exposure variation)—with strict identity and image separation from all major training datasets and controlled acquisition protocols. We propose an identity-isolation evaluation protocol, 10-fold generalization stability analysis, and a dedicated difficulty-aware evaluation paradigm. Contribution/Results: Experiments show that state-of-the-art models suffer significant accuracy drops (3–8%) on these new benchmarks compared to LFW, validating their effectiveness and necessity as next-generation evaluation standards for robust face verification.

Facial recognition faces privacy infringement and regulatory compliance risks—e.g., under GDPR—arising from real-world data collection. This work systematically evaluates the feasibility of synthetic facial data and, for the first time, proposes and empirically validates seven core privacy-preserving synthetic-data criteria, including identity leakage prevention, intra-class diversity, and inter-class separability. Leveraging multi-million-sample experiments, we conduct comprehensive evaluation across multiple benchmarks (e.g., CASIA-WebFace), assessing recognition accuracy, identity separation, intra-class variation, and fairness. Results show that the top-performing synthetic datasets—VariFace and VIGFace—achieve 95.67% and 94.91% accuracy, respectively, surpassing the real-world CASIA-WebFace benchmark (94.70%). Moreover, they enable controllable bias mitigation and ethically aligned generation. Our study establishes high-fidelity synthetic facial data as a scientifically sound, technically viable, and ethically necessary alternative paradigm.

This work proposes an efficient method for evaluating the intrinsic quality (IQ) of large-scale face recognition datasets without requiring full model training. By integrating neighborhood consistency scores with the effective rank of the embedding space, the approach establishes a lightweight, validation-free quality assessment framework capable of rapidly predicting downstream recognition performance using proxy models or dataset subsets. Experimental results demonstrate that the proposed IQ metric accurately forecasts model performance across clean, noisy, and mixed-quality datasets, substantially reducing the cost of data diagnosis and filtering. This provides a practical and scalable tool for preprocessing massive face datasets in real-world applications.

This study addresses the challenge that complex backgrounds in unconstrained scenarios—such as airport border control—degrade face recognition accuracy and impair the detection of presentation attacks. The authors systematically evaluate the impact of multiple face segmentation methods on four representative recognition models and three attack detection techniques through comprehensive experiments on datasets encompassing both controlled and unconstrained imagery. For the first time, they comprehensively demonstrate the dual role of background removal in simultaneously influencing recognition performance and security mechanisms, showing its significant effects on image quality, identification accuracy, and attack detectability. These findings provide empirical grounding and practical guidance for preprocessing strategies in real-world biometric systems, effectively bridging the critical gap between deployment feasibility and system reliability.

This study addresses the limitations of aggregate accuracy metrics in evaluating facial recognition systems within law enforcement contexts, which often obscure performance disparities across demographic groups and fail to capture true fairness and reliability. The authors propose moving beyond a single accuracy measure by introducing a fairness-aware evaluation framework coupled with a model-agnostic auditing strategy. By analyzing false positive and false negative rates at the subpopulation level, this approach uncovers hidden group-level biases that persist even when overall accuracy appears high. Empirical results demonstrate that systems with comparable aggregate accuracy can exhibit substantially different error distributions across demographic groups, underscoring the necessity and effectiveness of the proposed paradigm for enabling more responsible and equitable deployment of facial recognition technologies.

Deep learning models exhibit insufficient robustness against adversarial perturbations and common image corruptions, undermining their reliability in real-world deployment. To address this, we propose an active robustness verification strategy that leverages the training set itself: by performing local robustness analysis, our method automatically identifies “weakly robust” samples—serving as early, interpretable indicators of model vulnerability—and enables targeted robustness enhancement. Unlike conventional passive paradigms that rely solely on perturbed test sets for robustness evaluation, ours is the first to repurpose training data for robustness diagnostics. We integrate adversarial perturbation injection with diverse natural corruption tests. Extensive experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that our strategy significantly improves model robustness against both attacks and corruptions (average gain of +8.2%) while enhancing the sensitivity and interpretability of reliability assessment.