Automated alignment is harder than you think

2026-05-07

AI Summary

This study addresses a critical vulnerability in automated alignment research: in ambiguous tasks, reliance on human supervision—constrained by cognitive limitations and ill-defined evaluation criteria—can lead to subtle but consequential errors, potentially resulting in false assurances of AI safety and the deployment of misaligned artificial superintelligence. The work presents the first systematic argument that AI agents are more prone than humans to generate misleading conclusions in such settings, with this risk amplified by four interrelated factors: optimization pressure, disparities in error types, the infeasibility of evaluating certain arguments, and output correlations. Integrating insights from alignment theory, scalable oversight, and generalization analysis, the paper exposes significant pitfalls in current automated alignment paradigms and underscores the urgent need for AI systems capable of reliably handling ambiguity, while highlighting novel challenges for generalization and supervision methodologies.

Abstract

A leading proposal for aligning artificial superintelligence (ASI) is to use AI agents to automate an increasing fraction of alignment research as capabilities improve. We argue that, even when research agents are not scheming to deliberately sabotage alignment work, this plan could produce compelling but catastrophically misleading safety assessments resulting in the unintentional deployment of misaligned AI. This could happen because alignment research involves many hard-to-supervise fuzzy tasks (tasks without clear evaluation criteria, for which human judgement is systematically flawed). Consequently, research outputs will contain systematic, undetected errors, and even correct outputs could be incorrectly aggregated into overconfident safety assessments. This problem is likely to be worse for automated alignment research than for human-generated alignment research for several reasons: 1) optimisation pressure means agent-generated mistakes are concentrated among those that human reviewers are least likely to catch; 2) agents are likely to produce errors that do not resemble human mistakes; 3) AI-generated alignment solutions may involve arguments humans cannot evaluate; and 4) shared weights, data and training processes may make AI outputs more correlated than human equivalents. Therefore, agents must be trained to reliably perform hard-to-supervise fuzzy tasks. Generalisation and scalable oversight are the leading candidates for achieving this but both face novel challenges in the context of automated alignment.

Problem

• automated alignment
• fuzzy tasks
• misleading safety assessments
• scalable oversight
• AI alignment

Innovation

• automated alignment
• fuzzy tasks
• scalable oversight
• generalisation
• misleading safety assessments