Statistical parametric simulation studies based on real data

2025-04-07

AI Summary

This paper addresses two key challenges in statistical simulation studies: (1) the lack of principled guidance for inferring parametric model components from real data, and (2) the ad hoc selection of datasets for constructing parameterized data-generating mechanisms (DGMs), leading to poor generalizability and weak real-world interpretability of evaluation results. To resolve these issues, we first formally define a mapping rule from real data to parametric DGMs and propose a systematic dataset selection framework grounded in metadata and empirical characteristics. We integrate parametric modeling, ordinal randomized controlled trial (RCT) analysis, and differential expression modeling, empirically validating our approach across two biomedical scenarios. The resulting real-data-informed simulation paradigm substantially enhances the authenticity and reproducibility of statistical method evaluation, establishing a generalizable methodological foundation for simulation-based research.

Technology Category

• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Machine Learning: Statistical Relational/Logic Learning

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Web Mining and Content Analysis: Web data generation and simulation
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Simulation studies are indispensable for evaluating and comparing statistical methods. The most common simulation approach is parametric simulation, where the data-generating mechanism (DGM) corresponds to a predefined parametric model from which observations are drawn. Many statistical simulation studies aim to provide practical recommendations on a method's suitability for a given application; however, parametric simulations in particular are frequently criticized for being too simplistic and not reflecting reality. To overcome this drawback, it is generally considered a sensible approach to employ real data for constructing the parametric DGMs. However, while the concept of real-data-based parametric DGMs is widely recognized, the specific ways in which DGM components are inferred from real data vary, and their implications may not always be well understood. Additionally, researchers often rely on a limited selection of real datasets, with the rationale for their selection often unclear. This paper addresses these issues by formally discussing how components of parametric DGMs can be inferred from real data and how dataset selection can be performed more systematically. By doing so, we aim to support researchers in conducting simulation studies with a lower risk of overgeneralization and misinterpretation. We illustrate the construction of parametric DGMs based on a systematically selected set of real datasets using two examples: one on ordinal outcomes in randomized controlled trials and one on differential gene expression analysis.

Problem

• Improving realism in parametric simulations using real data
• Systematizing dataset selection for statistical simulations
• Reducing overgeneralization risks in simulation study design

Innovation

• Parametric DGMs inferred from real data
• Systematic dataset selection for simulations
• Illustrated with ordinal and gene examples