🤖 AI Summary
This study examines the practical validity of the official evaluation criteria used in Brazil’s CNPq Research Productivity (PQ) fellowship system, revealing significant discrepancies between stated standards and actual peer-review outcomes. By operationalizing policy dimensions into measurable variables and integrating CV data with OpenAlex metrics to construct predictive features, the research treats evaluation criteria as testable hypotheses for the first time. Employing an enhanced block-wise Boruta algorithm alongside interpretable machine learning models, the analysis achieves an average AUC of 0.96, indicating strong predictive power of PQ levels. However, only a few features—such as publication output, graduate student supervision, and administrative roles—demonstrate statistical significance, while several officially emphasized indicators contribute negligibly. These findings highlight a misalignment between formal evaluation criteria and real-world decision-making, underscoring the need for greater transparency in research assessment systems.
📝 Abstract
This paper empirically examines the practical validity of the official evaluation criteria underpinning the Research Productivity (PQ) Grant framework, as governed by the Brazilian National Council for Scientific and Technological Development (CNPq). By operationalizing regulatory dimensions (including bibliographic output, human resource training, and scientific recognition) as measurable variables extracted from CVs and OpenAlex bibliometric data, we treat policy-defined indicators as testable hypotheses rather than a priori assumptions. Using a block-based adaptation of the Boruta feature selection algorithm across several machine learning classifiers, we evaluate the statistical contribution of each dimension in distinguishing grant levels, with a focus on identifying top-tier (Level 1A) researchers. Our models achieve high predictive performance, with mean AUC scores reaching 0.96, indicating that PQ levels carry a robust and structured statistical signal. However, explanatory power is heavily concentrated within a limited subset of features, specifically bibliographic production, graduate-level supervision and institutional management roles. Conversely, several criteria explicitly emphasized in the regulations demonstrated no detectable statistical contribution to classification outcomes. These findings reveal a potential misalignment between the formal regulatory framework and the effective signals driving evaluation outcomes, suggesting that the practical evaluative signal is substantially more compact than officially stated and providing evidence-based insights for the refinement and transparency of research assessment policies.