🤖 AI Summary
Long-standing deficiencies in standardized, high-quality criteria for data science simulation studies have led to inconsistent design practices, poor reproducibility, and limited external validity. To address this, we propose MERITS—a simulation quality framework for trustworthy data science—systematically defining six core dimensions: Modularity, Efficiency, Realism, Stability, Intuitiveness, and Transparency. MERITS is the first to operationalize the PCS (Predictability-Computability-Stability) theory into concrete design principles and innovatively introduces a “cooking metaphor” to structure simulations as executable “recipes.” The framework includes 13 actionable design guidelines and is validated through empirical reconstruction of existing studies. Designed for cross-disciplinary applicability, MERITS has been successfully applied to diagnostic reconstructions of prior work, yielding substantial improvements in interpretability, reproducibility, and external validity.
📝 Abstract
Simulations play a crucial role in the modern scientific process. Yet despite (or due to) this ubiquity, the Data Science community shares neither a comprehensive definition for a"high-quality"study nor a consolidated guide to designing one. Inspired by the Predictability-Computability-Stability (PCS) framework for 'veridical' Data Science, we propose six MERITS that a simulation study should satisfy. (Modularity and Efficiency support the computability of a study, encouraging clean and flexible implementation. Realism and Stability address the conceptualization of the research problem: How well does a study predict reality, such that its conclusions generalize to new data/contexts? Finally, Intuitiveness and Transparency encourage good communication and trustworthiness of study design and results.) Drawing an analogy between simulation and cooking, we moreover offer (a) a conceptual framework for thinking about the anatomy of a simulation 'recipe'; (b) a baker's dozen in guidelines to aid the Data Science practitioner in designing one; and (c) a case study demonstrating the practical utility of our framework by using it to autopsy a preexisting simulation study. With this"PCS primer"for high-quality Data Science simulation, we seek to distill and enrich the best practices of simulation across disciplines into a cohesive recipe for trustworthy, veridical Data Science.