Trimming of extreme votes and favoritism: Evidence from the field

2026-02-05

AI Summary

This study addresses the lack of empirical evidence on national bias among reviewers in real-world peer review and how such bias is affected by extreme-score trimming mechanisms. Leveraging a large-scale dataset of 29,383 actual peer-review scores, the authors systematically evaluate the effectiveness of score trimming in mitigating home-country favoritism by comparing scoring patterns of multinational reviewers under conditions with and without the trimming mechanism. The findings reveal that reviewers exhibit significant favoritism toward applicants from their own country in the absence of trimming, but this bias is largely eliminated when the trimming mechanism is applied. These results suggest that extreme-score trimming can effectively reduce nationality-based bias in peer review processes.

Technology Category

Natural Language Processing: Ethics — Bias, Fairness, Transparency & PrivacyComputer Vision: Bias, Fairness & PrivacyKnowledge Representation and Reasoning: Preferences

Application Category

Social Networks and Social Media: Fairness and bias in social network and social media analysisSearch and Retrieval-Augmented AI: Web evaluation methodologies and metricsUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Despite a large body of theoretical literature on voting mechanisms, there is no documented evidence from real-world panel evaluations about the effect of trimming the extreme votes on sincere voting. We provide the first such evidence by comparing subjective evaluations of experts from different countries in competitive settings with and without a trimming mechanism. In these evaluations, some of the evaluated subjects are experts'compatriots. Using data on 29,383 subjective evaluations, we find that experts assign significantly higher scores to their compatriots in panels without trimming. However, in panels with trimming, this favoritism is generally insignificant.

Problem

• trimming
• favoritism
• sincere voting
• panel evaluation
• extreme votes

Innovation

• vote trimming
• favoritism
• sincere voting
• panel evaluation
• expert bias