Behind the Screens: Uncovering Bias in AI-Driven Video Interview Assessments Using Counterfactuals

2025-05-17

AI Summary

AI-driven video interview personality assessment—grounded in the Big Five model—is vulnerable to training data biases, yielding discriminatory outcomes across protected attributes such as gender, race, and age. Method: We propose a counterfactual fairness auditing framework that requires no access to the black-box model or its training data. It introduces the first counterfactual generation and bias quantification method tailored to audiovisual–text multimodal interview data. Leveraging GANs for cross-modal feature alignment and validated via protected-attribute classifiers, the framework generates counterfactuals that preserve personality predictions while perturbing sensitive attributes. Contribution/Results: Empirical evaluation reveals significant group-level bias in state-of-the-art personality prediction models. Counterfactual samples reduce protected-attribute classification accuracy to below 15%, confirming generation fidelity and fairness intervention efficacy. The framework enables scalable, third-party fairness auditing of proprietary recruitment systems without model access—delivering the first plug-and-play bias detection tool for commercial AI hiring platforms.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Philosophy and Ethics of AI: Bias, Fairness & Equity

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems

Abstract

AI-enhanced personality assessments are increasingly shaping hiring decisions, using affective computing to predict traits from the Big Five (OCEAN) model. However, integrating AI into these assessments raises ethical concerns, especially around bias amplification rooted in training data. These biases can lead to discriminatory outcomes based on protected attributes like gender, ethnicity, and age. To address this, we introduce a counterfactual-based framework to systematically evaluate and quantify bias in AI-driven personality assessments. Our approach employs generative adversarial networks (GANs) to generate counterfactual representations of job applicants by altering protected attributes, enabling fairness analysis without access to the underlying model. Unlike traditional bias assessments that focus on unimodal or static data, our method supports multimodal evaluation-spanning visual, audio, and textual features. This comprehensive approach is particularly important in high-stakes applications like hiring, where third-party vendors often provide AI systems as black boxes. Applied to a state-of-the-art personality prediction model, our method reveals significant disparities across demographic groups. We also validate our framework using a protected attribute classifier to confirm the effectiveness of our counterfactual generation. This work provides a scalable tool for fairness auditing of commercial AI hiring platforms, especially in black-box settings where training data and model internals are inaccessible. Our results highlight the importance of counterfactual approaches in improving ethical transparency in affective computing.

Problem

• Detecting bias in AI-driven personality assessments for hiring
• Evaluating fairness using counterfactuals without model access
• Analyzing multimodal bias in visual, audio, and textual features

Innovation

• Uses GANs for counterfactual applicant representations
• Supports multimodal bias evaluation across features
• Enables fairness auditing in black-box AI systems