Understanding the decision-making process of choice modellers

2024-11-03

arXiv.org

AI Summary

This study investigates how methodological choices made by modelers during discrete choice model construction—particularly in noise pollution policy contexts—induce substantial variation in outcomes. Method: We develop the “Serious Choice Modeling Game,” a behavioral experiment platform that systematically tracks modelers’ data exploration, model specification, and interpretation processes, integrating operational logs, descriptive statistics, and multinomial logit modeling for quantitative analysis. Contribution/Results: We find that while data visualization is widespread, missing-data handling is frequently neglected; parsimony preferences and iteration quality significantly affect model fit and simplicity; and substantial heterogeneity exists across modeling strategies applied to identical data. Crucially, willingness-to-pay estimates vary markedly with methodological choices, undermining policy recommendation reliability. This work provides the first systematic, causal evidence linking modeler behavior to outcomes in policy-oriented choice models, offering empirical foundations for enhancing reproducibility and policy robustness of discrete choice analysis.

Technology Category

• Game Theory and Economic Paradigms: Social Choice / Voting
• Knowledge Representation and Reasoning: Preferences
• Machine Learning: Learning Preferences or Rankings

Application Category

• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Discrete Choice Modelling serves as a robust framework for modelling human choice behaviour across various disciplines. Building a choice model is a semi structured research process that involves a combination of a priori assumptions, behavioural theories, and statistical methods. This complex set of decisions, coupled with diverse workflows, can lead to substantial variability in model outcomes. To better understand these dynamics, we developed the Serious Choice Modelling Game, which simulates the real world modelling process and tracks modellers' decisions in real time using a stated preference dataset. Participants were asked to develop choice models to estimate Willingness to Pay values to inform policymakers about strategies for reducing noise pollution. The game recorded actions across multiple phases, including descriptive analysis, model specification, and outcome interpretation, allowing us to analyse both individual decisions and differences in modelling approaches. While our findings reveal a strong preference for using data visualisation tools in descriptive analysis, it also identifies gaps in missing values handling before model specification. We also found significant variation in the modelling approach, even when modellers were working with the same choice dataset. Despite the availability of more complex models, simpler models such as Multinomial Logit were often preferred, suggesting that modellers tend to avoid complexity when time and resources are limited. Participants who engaged in more comprehensive data exploration and iterative model comparison tended to achieve better model fit and parsimony, which demonstrate that the methodological choices made throughout the workflow have significant implications, particularly when modelling outcomes are used for policy formulation.

Problem

• Understanding variability in choice model outcomes due to diverse workflows
• Analyzing modellers' decisions in handling data and model specification
• Exploring preference for simpler models despite complex alternatives

Innovation

• Serious Choice Modelling Game simulates real-world process
• Tracks modellers' decisions using stated preference dataset
• Analyzes individual decisions and modelling approach differences