Accounting for Context: Shaping Moral Credences for Value Alignment

2026-06-05

AI Summary

This work addresses a critical limitation in existing moral aggregation methods, which often neglect contextual factors and thereby risk biases analogous to Simpson’s paradox. By incorporating context-sensitivity into decision-making under moral uncertainty, the paper formally models how diverse moral theories are aggregated across varying situations and systematically examines the relationship between such contextualized aggregation and the weak Pareto principle. The analysis reveals that once contextual factors are accounted for, conventional aggregation mechanisms may violate the weak Pareto principle, exposing their inherent limitations. Integrating insights from context-sensitive decision theory with formal ethical reasoning, this study advances a more robust modeling paradigm for value alignment in artificial agents.

Abstract

Ensuring that agent behaviours are aligned with human moral values inevitably raises the problem of how to account for the plurality of moral perspectives that societies -- and even individuals -- typically adopt. Work on moral uncertainty proposes mechanisms to fairly and democratically aggregate evaluations of actions across different moral theories. However, this paper argues that one needs to account for contextual factors when aggregating moral evaluations. For example, consequentialist perspectives assume an ability to accurately determine how an agent's actions change the world; an assumption that often does not hold in real world settings. We, therefore, formalise agent decision making under moral uncertainty, while also accounting for these kinds of contextual factors. We thereby show that a seemingly commonsensical property -- the weak Pareto principle -- is violated. We argue that this apparent problem is, in fact, a variation of Simpson's paradox, and hence reveals the limitations of aggregation mechanisms that ignore the impact of contextual factors.

Problem

• moral uncertainty
• value alignment
• contextual factors
• moral aggregation
• Simpson's paradox

Innovation

• moral uncertainty
• contextual factors
• value alignment
• Simpson's paradox
• weak Pareto principle