🤖 AI Summary
This study addresses the risk of misallocating capital and authority when inferring individual skill from outcome records in high-noise, low-effective-sample decision contexts. The authors propose a two-dimensional evaluation framework based on outcome noise intensity and the number of effective independent observations, revealing that prevalent performance assessments—such as those for mutual funds, venture capital, and executive evaluations—typically fall within regions where skill attribution is statistically unreliable. To mitigate this, they adapt causal validation logic from medical research, integrating statistical inference with effective sample size estimation to quantify how noise distorts skill assessment. When signal strength is insufficient, the paper advocates replacing individual-level attribution with group-level empirical methods, offering a more robust approach to evaluation in high-stakes decision domains.
📝 Abstract
When do outcome records carry enough signal to support reliable inferences about skill? When they do not, what should evaluators substitute? The framework answering the first question characterizes any decision domain with two parameters: the noise reflected in each outcome and the effective number of independent outcomes that are available over an observation window. When domains are positioned in a two-dimensional space of noise versus number of outcomes, those in which capital, prestige, and political power are routinely allocated on the basis of realized outcomes (e.g., mutual fund management, venture capital, executive performance) fall in the region where outcome records contain too little signal to support reliable individual-level inferences. Evaluating actors when outcome records are insufficient can be done by adopting the populationlevel empirical validation methods long used in medicine: has the actor adopted the practices that, at the population level, are associated with better outcomes?