Measuring Gender Bias in Job Title Matching for Grammatical Gender Languages

2025-09-17

AI Summary

This study investigates how explicit gender marking in occupational titles within grammatical-gender languages induces systemic gender bias in automated ranking systems. We propose a quantitative evaluation framework grounded in rank-similarity metrics—particularly Rank-Biased Overlap (RBO)—and introduce the first publicly available, multilingual occupational-title matching benchmark covering four grammatical-gender languages (Spanish, French, German, Russian), annotated with fine-grained gender and relevance labels. Using multilingual pretrained models on cross-lingual title-matching tasks, we demonstrate that all mainstream off-the-shelf models exhibit consistent, statistically significant gender bias in ranking outputs. These findings reveal the profound impact of grammatical gender on ranking fairness. Our work establishes a new methodological paradigm and benchmark for assessing gender bias in occupational ranking, enabling rigorous, reproducible evaluation across languages and models.

Technology Category

• Machine Learning: Learning Preferences or Rankings
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy
• Computer Vision: Bias, Fairness & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Social Networks and Social Media: Fairness and bias in social network and social media analysis

Abstract

This work sets the ground for studying how explicit grammatical gender assignment in job titles can affect the results of automatic job ranking systems. We propose the usage of metrics for ranking comparison controlling for gender to evaluate gender bias in job title ranking systems, in particular RBO (Rank-Biased Overlap). We generate and share test sets for a job title matching task in four grammatical gender languages, including occupations in masculine and feminine form and annotated by gender and matching relevance. We use the new test sets and the proposed methodology to evaluate the gender bias of several out-of-the-box multilingual models to set as baselines, showing that all of them exhibit varying degrees of gender bias.

Problem

• Measuring gender bias in job title matching systems
• Evaluating multilingual models for grammatical gender bias
• Developing test sets for gender-controlled ranking evaluation

Innovation

• Proposes RBO metrics for gender bias evaluation
• Generates gender-annotated test sets for four languages
• Evaluates multilingual models' gender bias using new methodology