AI Coding Agents Can Reproduce Social Science Findings

2026-06-09

AI Summary

This work addresses the lack of a reliable, large-scale evaluation benchmark for AI coding agents in social science replication tasks and the difficulty in disentangling agent capability from the quality of source materials. The authors propose SocSci-Repro-Bench, a novel benchmark comprising 221 replication tasks spanning four disciplines and thirteen subfields, which explicitly separates reproducible from non-reproducible tasks to enable problem-isolated evaluation at scale. Experiments reveal that Claude Code substantially outperforms Codex, with both models achieving far higher replication success rates than general-purpose large language models. Agents demonstrate accurate comprehension of research questions, and performance gains are primarily attributable to reasoning capabilities rather than memorization. Furthermore, prompt phrasing can induce confirmatory analytical behavior, while providing the original paper as a PDF yields only marginal improvements yet introduces bias.

Abstract

Recent anecdotal evidence suggests that AI coding agents can reproduce published findings when provided with original data and code; yet systematic evaluation across social sciences remains limited. Existing evaluation benchmarks are insufficient, either small or conflate agent performance with problems in the reproduction materials themselves, such as code that fails to execute correctly. Here we introduce SocSci-Repro-Bench, a benchmark of 221 tasks spanning four disciplines and 13 substantive domains, constructed from studies whose results are either fully reproducible with available materials or demonstrably non-reproducible due to missing data, allowing us to isolate agents' reproduction capacity. Evaluating two frontier coding agents, Claude Code and Codex, we find that both can reproduce a large share of social science findings, with Claude Code substantially outperforming Codex. These reproduction rates considerably exceed those previously reported for general-purpose LLM-based agents on comparable reproducibility benchmarks. Both agents also perform strongly on a reasoning task requiring identification of underlying research questions, and additional analyses suggest that results are not primarily driven by memorization. Providing the original paper PDF alongside replication materials modestly improves performance but introduces bias on tasks where reproduction is impossible. We also show that agents can be nudged toward confirmatory specification search through subtle prompt framing. Together, these findings suggest that at least some frontier coding agents can serve as reliable executors of computational workflows while underscoring the need for careful benchmarking and prompt design as AI systems assume larger roles in scientific production.

Problem

• AI coding agents
• reproducibility
• social science
• benchmarking
• computational workflows

Innovation

• reproducibility benchmark
• AI coding agents
• social science replication
• prompt design
• computational workflow automation