The Emotional Alignment Design Policy

2025-07-07

AI Summary

This paper addresses the misalignment between affective responses elicited by AI entities and their actual capabilities or moral status—manifesting as over-anthropomorphism, affective mismatch, or insufficient emotional responsiveness. To address this, we propose an “affective alignment” design ethics framework—the first to systematically distinguish between intensity mismatches and categorical mismatches in human-AI affective interaction. Drawing on normative ethics, human-computer interaction, and cognitive science, the framework employs normative analysis and scenario-based reasoning to tackle core challenges: respecting user autonomy, reconciling expert-public value divergences, and navigating uncertainty regarding AI’s moral standing. Its primary contribution is a rigorously grounded, actionable ethical guideline system that equips designers of AI and robotic systems with theoretical foundations and practical pathways to mitigate affective deception, enhance moral transparency, and foster relationally trustworthy human-AI interactions.

Technology Category

• Philosophy and Ethics of AI: Morality & Value-based AI
• Humans and AI: Learning Human Values and Preferences
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Accountability, Transparency, and Ethics for personalization
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

According to what we call the Emotional Alignment Design Policy, artificial entities should be designed to elicit emotional reactions from users that appropriately reflect the entities' capacities and moral status, or lack thereof. This principle can be violated in two ways: by designing an artificial system that elicits stronger or weaker emotional reactions than its capacities and moral status warrant (overshooting or undershooting), or by designing a system that elicits the wrong type of emotional reaction (hitting the wrong target). Although presumably attractive, practical implementation faces several challenges including: How can we respect user autonomy while promoting appropriate responses? How should we navigate expert and public disagreement and uncertainty about facts and values? What if emotional alignment seems to require creating or destroying entities with moral status? To what extent should designs conform to versus attempt to alter user assumptions and attitudes?

Problem

• Designing AI to evoke appropriate emotional responses from users
• Balancing user autonomy with promoting correct emotional reactions
• Addressing ethical dilemmas in creating emotionally aligned artificial entities

Innovation

• Design artificial entities for appropriate emotional reactions
• Balance user autonomy with promoting correct responses
• Address moral status in emotional alignment design