The illusion of neutrality in metric-based research evaluation

2026-09-21

AI Summary

研究探讨了科研评价中场地声望与引用数之间的权衡问题，通过调查869名研究人员对假设招聘规则的选择偏好来揭示个人差异及主观判断的影响。

Abstract

Venue prestige and citation counts are two widely used, albeit imperfect, signals of research quality. When the two signals conflict, evaluators must decide how much weight to assign each. Yet, it remains unknown how researchers across disciplines trade off these two signals when evaluating research outcome. To fill this gap, we surveyed 869 researchers using paired choices between hypothetical departmental hiring rules that assigned different weights to venue prestige and citation counts, asking which would produce better science. Among 795 respondents whose choices were largely internally consistent, choices placed nearly equal aggregate weight on the two signals. This apparent balance concealed substantial individual heterogeneity: nearly two in five respondents occupied the most venue-heavy or citation-heavy intervals. Moreover, respondents on average chose more citation-heavy rules than they believed their departments used in hiring. By revealing the subjective judgments that arise when research indicators conflict, our findings reinforce calls for greater caution when quantitative indicators are used to evaluate research.

Problem

• venue prestige
• citation counts
• research evaluation

Innovation

• venue prestige
• citation counts
• research evaluation
• subjective judgments