Empirical Decision Theory

2025-12-05

AI Summary

Conventional decision theory relies on the closed-world or small-world assumption, requiring explicit enumeration of unobservable world states—rendering it inapplicable when such states are inaccessible. Method: We propose a fully state-free decision model grounded solely in observable action–outcome sequences. Optimality is defined via an empirical selection function, eliminating any need to model latent world states. Our approach integrates statistical estimation, consistency-based hypothesis testing, and direct inference from confidence sets to ensure statistical consistency and robustness. Contribution/Results: This is the first decision-theoretic framework whose primitive elements consist exclusively of action–outcome sequences. Theoretical analysis demonstrates its effectiveness in comparatively evaluating prompting strategies in generative AI, offering both strong practical applicability and methodological originality.

Technology Category

• Reasoning under Uncertainty: Sequential Decision Making
• Search and Optimization: Adversarial Search
• Humans and AI: Planning and Decision Support for Human-Machine Teams

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Analyzing decision problems under uncertainty commonly relies on idealizing assumptions about the describability of the world, with the most prominent examples being the closed world and the small world assumption. Most assumptions are operationalized by introducing states of the world, conditional on which the decision situation can be analyzed without any remaining uncertainty. Conversely, most classical decision-theoretic approaches are not applicable if the states of the world are inaccessible. We propose a decision model that retains the appeal and simplicity of the original theory, but completely overcomes the need to specify the states of the world explicitly. The main idea of our approach is to address decision problems in a radically empirical way: instead of specifying states and consequences prior to the decision analysis, we only assume a protocol of observed act--consequence pairs as model primitives. We show how optimality in such empirical decision problems can be addressed by using protocol-based empirical choice functions and discuss three approaches for deriving inferential guarantees: (I) consistent statistical estimation of choice sets, (II) consistent statistical testing of choice functions with robustness guarantees, and (III) direct inference for empirical choice functions using credal sets. We illustrate our theory with a proof-of-concept application comparing different prompting strategies in generative AI models.

Problem

• Develops a decision model without requiring explicit world states
• Addresses uncertainty using empirical act-consequence observation protocols
• Provides inferential guarantees via statistical estimation and testing methods

Innovation

• Empirical decision model without specifying world states
• Uses protocol-based empirical choice functions for optimality
• Derives inferential guarantees via statistical estimation and testing