COARA will not save science from the tyranny of administrative evaluation

2024-08-10

arXiv.org

AI Summary

This paper critically examines the Coalition for Advancing Research Assessment (CoARA) reform, arguing that its administrative governance of peer review and responsibility-based metrics perpetuates epistemic and institutional authoritarianism—concentrating quality-definition authority while excluding public deliberation, thereby undermining the democratic ideal of “well-ordered science.” Method: Drawing on Philip Kitcher’s normative framework of well-ordered science for the first time in this context, the study integrates philosophical analysis, critical sociology of science, and normative political theory to diagnose CoARA’s structural deficiencies in procedural fairness and democratic legitimacy. Contribution/Results: It demonstrates that administrative evaluation must be strictly confined to essential domains (e.g., hiring, funding) and embedded within robust fair procedures and inclusive public deliberation. The findings yield an ethical roadmap for research assessment reform centered on democratic accountability and epistemic justice.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Knowledge Representation and Reasoning: Argumentation
• Reasoning under Uncertainty: Causality

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Responsible Web: Algorithmic accountability and transparency on the web

Abstract

The Coalition for Advancing Research Assessment (CoARA) agreement is a cornerstone in the ongoing efforts to reform research evaluation. CoARA advocates for administrative evaluations of research that rely on peer review, supported by responsible metrics, as beneficial for both science and society. Its principles can be critically examined through the lens of Philip Kitcher's concept of well-ordered science in a democratic society. From Kitcher's perspective, CoARA's approach faces two significant challenges: definitions of quality and impact are determined by governments or evaluation institutions rather than emerging from broad public deliberation, and a select group of scientists is empowered to assess research based on these predefined criteria. This creates susceptibility to both the ''tyranny of expertise'' and the ''tyranny of ignorance'' that Kitcher cautions against. Achieving Kitcher's ideal would require limiting administrative evaluations to essential tasks, such as researcher recruitment and project funding, while establishing procedures grounded in principles of fairness.

Problem

• Scientific Evaluation
• Fairness in Assessment
• Expert Dominance

Innovation

• CoARA Protocol
• Balanced Expert-Lay Participation
• Fair Evaluation Framework