governance design

Designing technical, legal, and operational mechanisms (consent, traceability, accountability, dispute resolution, explainability) to govern deployment, oversight, and ethical use of AI systems and agent identifiers within organizations and socio-technical systems.

governancedesign

12-Month Skill Trend

Momentum and market value over time
Trending
Score
+20 in 12 mo
96
12 mo agoNow
Career
Value
+$12K in 12 mo
$42K/year
12 mo agoNow

Recommended Survey Paper

Quick overview of the field
View more

Must-Read Papers

Most classic and influential ideas
View more

Towards an AI Accountability Policy

Jul 25, 2023
PG
Przemyslaw Grabowicz
🏛️ University of Massachusetts Amherst

Regulatory oversight of high-risk AI systems suffers from insufficient explainability and a lack of standardized benchmarking, leading to accountability gaps and eroded stakeholder trust. Method: This study establishes a tiered explainability framework and benchmark evaluation system, introducing the first use-case-specific AI compliance certification mechanism for anticipated high-risk applications. Drawing on regulatory precedents from the U.S. FDA and the EU AI Act, it proposes a dedicated AI regulatory authority and mandates standardized internal auditing, fairness-aware statistical assessment, and input impact analysis. We develop automated auditing tools, a structured benchmarking framework, a compliance certificate generation system, and a public AI registry. Contribution/Results: The work delivers an actionable, risk-based governance blueprint that significantly enhances regulatory efficiency and user trust. It provides critical technical infrastructure and operational templates for international regulatory frameworks—including the EU AI Act—enabling scalable, evidence-based AI oversight.

Establish oversight for AI systemsPromote accountability in high-risk AIStandardize explainability and benchmarking

Oversight Structures for Agentic AI in Public-Sector Organizations

Jun 05, 2025
CS
Chris Schmitz
🏛️ Hertie School | University of Oxford | Weizenbaum Institute | Technical University Munich

This paper addresses the regulatory failure exacerbated by deploying autonomous AI—particularly embodied agents—in the public sector, where traditional siloed, stage-gated approval mechanisms fail to meet three emerging needs: continuous oversight, deep integration of governance into operational workflows, and cross-agency coordination. Adopting a mixed-methods approach—systematic literature review complemented by in-depth interviews with frontline public officials—the study identifies, for the first time, five core AI governance dimensions tailored to public-sector contexts: cross-agency implementation, holistic assessment, enhanced security, operational transparency, and systemic auditing. Based on these, it proposes a novel “agent-oriented regulatory framework” that is institutionally adaptive and technically interoperable. The framework bridges theory and practice, offering actionable guidance for governing autonomous AI systems under real-world institutional constraints—thereby filling a critical gap in the literature on public-sector AI regulation.

Addressing intensified oversight challenges from agentic AI in public sectorsIdentifying five key governance dimensions for responsible AI deploymentProposing adapted institutional structures for public-sector AI oversight

This study addresses a critical gap in current AI governance, which predominantly emphasizes substantive rules while neglecting the legal and regulatory infrastructure necessary for their generation and implementation. For the first time, this work systematically positions legal infrastructure as the cornerstone of effective AI governance and proposes an institutional framework comprising a frontier model registration system, an autonomous agent identification mechanism, and a market-oriented regulatory service model. Through rigorous legal design, regulatory modeling, and policy mechanism analysis, the research delivers an actionable institutional pathway that significantly enhances the flexibility, scalability, and enforcement efficacy of AI governance rules.

AI governancelegal infrastructureregulatory framework

Open Problems in Technical AI Governance

Jul 20, 2024
AR
Anka Reuel
🏛️ Stanford University | Centre for the Governance of AI | Oxford Martin AI Governance Initiative | MIT CSAIL | Institute for Progress | Center for a New American Security | interface – Tech Analysis and Policy Ideas for Europe e.V. | Institute for AI Policy and Strategy | University of Oxford | Cooperative AI Foundation | Mila | OpenMined | Cohere For AI | Hugging Face | University of Cambridge | The Future Society | University of California, Berkeley | University of Montreal | MIT | Stanford HAI | Palisade R

Rapid AI advancement poses novel governance challenges, necessitating a rigorous, technically grounded approach to AI governance. Method: This work introduces “technical AI governance” as a distinct paradigm and establishes the first interdisciplinary analytical framework—integrating AI safety, mechanism design, policy modeling, and governance theory—to systematically address three core problem domains: risk identification, evaluation of intervention effectiveness, and compliance mechanism design. Adopting a problem-driven methodology, it clarifies how technical tools can concretely support governance practice. Contributions/Results: (1) A formal, structured definition of technical AI governance and a taxonomy of its core problems; (2) The first publicly available, extensible open-problems catalog for technical AI governance, bridging methodological gaps between technical and policy communities; and (3) An actionable, problem-oriented investment guide for researchers and funding agencies to prioritize high-impact technical governance research.

Address technical barriers in AI governanceDevelop mechanisms for AI enforcement and complianceIdentify and assess effective governance actions

This study addresses the challenge of assigning legal responsibility in highly autonomous AI systems that lack legal personhood, rendering traditional human-centric liability frameworks inadequate. The authors propose “Operational Agency”—a penetrable legal fiction—coupled with the “Operational Agency Graph” (OAG), a causal modeling tool that maps accountability chains in human-AI collaboration by analyzing the AI’s goal-directedness, predictive capacity, and safety architecture. This framework uniquely introduces Operational Agency as a post-hoc evidentiary mechanism within legal doctrine, integrating principles from corporate criminal liability, the innocent agent doctrine, and vicarious liability—without conferring legal personhood on AI. Validated across five real-world scenarios, including autonomous vehicle accidents and algorithmic collusion, the approach offers courts, legislators, and regulators a principled foundation for accountability that reconciles technical autonomy with human responsibility.

actus reusartificial intelligenceculpability

Latest Papers

What's happening recently
View more

This study addresses critical misalignments between current legal frameworks and the technical realities of AI agents. Existing regulations often conflate model capabilities with agent architectures, misinterpret probabilistic predictions as cognitive decisions, and misunderstand the nature of autonomy, thereby failing to effectively govern the actual operational mechanisms of AI systems. Through a systematic comparison of eleven regulatory texts—including the EU AI Act, OECD/G7 principles, and NIST guidelines—against six prevalent AI agent development architectures, this work employs literature analysis, cross-regulatory comparison, and architectural dissection to uncover structural discrepancies between legal definitions and technical implementations. It proposes, for the first time, a consensus-based technical definition grounded in developer documentation that explicitly incorporates key autonomy-enabling elements such as system prompts, API permissions, sandboxing mechanisms, and orchestration code, offering both a theoretical foundation and a practical framework for precise and effective AI regulation.

agentic AIAI autonomylegal frameworks

This work addresses the inadequacy of existing legal frameworks in effectively governing emergent networked environments composed of autonomous AI agents, particularly highlighting critical gaps in identity, authorization, and accountability mechanisms. To bridge these gaps, the paper proposes a Distributed Legal Infrastructure (DLI) grounded in a five-layer interlocking architecture: self-sovereign soulbound identities, AI cognitive constraints, decentralized dispute resolution, insurance-based market regulation, and a portable institutional framework. This design embeds legality intrinsically within distributed AI systems, establishing a governance foundation that ensures agent accountability, enables contestability, and aligns with the rule of law. By doing so, the DLI facilitates the compliant evolution of AI-driven societies while supporting cross-system legal interoperability.

accountabilityagentic webAI agents

This work addresses the absence of standardized, composable oversight infrastructure in current AI deployments, which leads teams to repeatedly build fragmented auditing and monitoring mechanisms. The authors propose a five-layer, six-dimension framework for AI oversight, with a particular focus on formally defining— for the first time—the “normative layer.” This layer translates human intent into executable, traceable, and upgradable machine-checkable norms through six design principles, including elicitable, adversarially aware, and governable specifications. Integrating formal methods, policy languages (e.g., Cedar, OPA), and constitutional AI concepts, the study introduces CARMA, a norm-driven runtime oversight prototype that demonstrates how a single norm can uniformly drive execution, evaluation, and upgrading. The system validates the feasibility of reusing composable oversight components across teams.

AI OversightComposabilityCoordination Gap

This study addresses the fundamental mismatch between current AI agents and human-centric identity paradigms, stemming from AI’s lack of embodiment, persistent memory, and legal personhood. Through structured comparative analysis, regulatory assessment, and identity lifecycle modeling, the work reveals a foundational asymmetry between humans and AI across four dimensions: substrate, persistence, verifiability, and legal status. It argues that directly applying conventional identity frameworks to AI leads to systemic failure. The research identifies five critical structural gaps—semantic intent verification, recursive delegation accountability, identity integrity, governance transparency and enforcement mechanisms, and operational sustainability—thereby establishing a theoretical foundation for designing novel identity architectures tailored to the unique characteristics of artificial intelligence.

accountabilityAI Identityautonomous agents

This study addresses the challenges posed by autonomous or semi-autonomous AI contributors to human-centric open-source governance mechanisms, which have led to policy fragmentation and misalignment with emerging AI regulations. Employing a most-similar systems design, the research combines policy text analysis, indicator coding, and process tracing to comparatively examine AI contribution policies across six open-source organizations. It proposes the first six-dimensional governance taxonomy and a policy maturity scoring framework specifically tailored for open-source AI contributions. The analysis identifies critical governance failures in dimensions such as disclosure, accountability, and oversight, and reveals significant coordination gaps between existing policies and international AI regulatory standards. Building on these findings, the study outlines an initial, calibratable, tiered coordination governance framework to bridge these disconnects.

AI contributorsAI governanceopen source software

Hot Scholars

SS

Sharifa Sultana

Assistant Professor, Computer Science, University of Illinois Urbana-Champaign
HCIResponsible AIDesign
RM

Rashid Mushkani

University of Montreal I Mila
Public (Space & Life)Sociotechnical AIUrban AnalyticsCommunity-Centered AI
AX

Amy X. Zhang

Associate Professor, Computer Science & Engineering, University of Washington
social computingHCI
EP

Evangelos Pournaras

Professor of Trustworthy Distributed Intelligence, UKRI Future Leaders Fellow, University of Leeds
distributed intelligenceblockchaincollective decision-makingdigital democracy
AK

Atoosa Kasirzadeh

Carnegie Mellon University
AI EthicsAI GovernancePhilosophyMathematical Optimization