pAI-Econ-claude: A Gated Human-in-the-Loop Multi-Agent Architecture for AI-Assisted Economic Theory Development

2026-07-23

AI Summary

In social science research, where cheap, complete, and machine-readable correctness signals are often unavailable, existing AI systems struggle to ensure result reliability. This work proposes a gated human-in-the-loop multi-agent architecture that coordinates agent collaboration through a shared workspace, incorporates a dedicated gating module to diagnose failure modes and trigger backtracking, and embeds human judgment at critical irreversible decision points to retain ultimate control. Departing from paradigms that rely on agents’ self-certification of correctness, the design prioritizes process auditability and clarifies human–AI collaboration boundaries. Experiments across five economic theory tasks demonstrate that, compared to a non-gated baseline, the approach reduces average failure severity from 1.58 to 1.16 and improves overall usefulness from 2.60 to 3.10, with consistent evaluator preference in four of the five tasks.

Abstract

In many social-science research tasks, such as economics, LLM-based agents must produce outputs for which no cheap, task-complete, machine-readable correctness signal exists. This creates a distinctive reliability problem for multi-agent systems: how should generation, critique, coordination, and human judgment be organized when no component can certify the final result? We address this problem through pAI-Econ-claude, a gated, human-in-the-loop multi-agent architecture for AI-assisted economic theory development. Agents coordinate through a shared workspace of inspectable intermediate records; specialized gates diagnose targeted failure modes and recommend loopbacks without certifying correctness; and human checkpoints retain authority over decisions that are costly to reverse. We evaluate the architecture on five matched economic-theory tasks against an ungated baseline. Two evaluators blinded to configuration agreed on all five pairwise rankings, preferring the gated architecture in four tasks and the baseline in one. Mean failure severity fell from 1.58 to 1.16, while overall usefulness rose from 2.60 to 3.10. The largest observed gain occurred when a reality check rejected a false market-structure premise and a proof review prompted revision of a false welfare claim. The negative case shows that scaffolding can also compress an economically important mechanism too aggressively. The results support a bounded claim: gated oversight improves the auditability of AI-assisted economic theory without substituting for formal verification, and the allocation of irreversible human judgment is a more informative design variable than pure agent autonomy. The workflow is publicly available at https://github.com/maxwell2732/pAI-Econ-claude.

Problem

• reliability
• human-in-the-loop
• multi-agent systems
• economic theory
• correctness signal

Innovation

• gated architecture
• human-in-the-loop
• multi-agent system
• auditability
• economic theory development