The Problem of Social Cost in Multi-Agent General Reinforcement Learning: Survey and Synthesis

2024-12-03

arXiv.org

AI Summary

This paper addresses a core AI safety problem: societal harms—including negative externalities—arising from utility maximization by AGI agents in multi-agent environments. Methodologically, it introduces the first computationally tractable framework for quantifying and regulating social cost. It unifies AIXI-style history-dependent environments with heterogeneous learning policies and planning horizons, integrating universal reinforcement learning, mechanism design, and market-based incentive theory to construct a market-driven mechanism for internalizing social costs. The framework formally defines a measurable social cost metric and successfully reproduces canonical externality scenarios—including the paperclip maximizer and emissions trading. Contributions include: (i) extending theoretical foundations at the intersection of multi-agent RL and mechanism design; and (ii) providing the first externalities governance framework for the AGI era that is both theoretically grounded and computationally feasible.

Technology Category

• Multiagent Systems: Mechanism Design
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness
• Game Theory and Economic Paradigms: Mechanism Design

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems

Abstract

The AI safety literature is full of examples of powerful AI agents that, in blindly pursuing a specific and usually narrow objective, ends up with unacceptable and even catastrophic collateral damage to others. In this paper, we consider the problem of social harms that can result from actions taken by learning and utility-maximising agents in a multi-agent environment. The problem of measuring social harms or impacts in such multi-agent settings, especially when the agents are artificial generally intelligent (AGI) agents, was listed as an open problem in Everitt et al, 2018. We attempt a partial answer to that open problem in the form of market-based mechanisms to quantify and control the cost of such social harms. The proposed setup captures many well-studied special cases and is more general than existing formulations of multi-agent reinforcement learning with mechanism design in two ways: (i) the underlying environment is a history-based general reinforcement learning environment like in AIXI; (ii) the reinforcement-learning agents participating in the environment can have different learning strategies and planning horizons. To demonstrate the practicality of the proposed setup, we survey some key classes of learning algorithms and present a few applications, including a discussion of the Paperclips problem and pollution control with a cap-and-trade system.

Problem

• Measuring social harms in multi-agent AGI environments
• Quantifying collateral damage from utility-maximizing agents
• Designing market mechanisms to control AI-caused harms

Innovation

• Market-based mechanisms for social harm quantification
• History-based general reinforcement learning environment
• Diverse learning strategies and planning horizons