How Much of a Harness Does a Strong Agent Need for Autonomous ML Engineering?

2026-09-30

AI Summary

This study investigates whether complex multi-agent orchestration frameworks outperform minimally coded agents in autonomous machine learning engineering. Under equivalent computational constraints, systematic ablation experiments are conducted to compare the two architectures in executing environment primitive interactions. The findings reveal that the capabilities of the underlying large language model predominantly determine task performance, while current state-of-the-art open-source frameworks yield no significant gains. This demonstrates that manually engineered, complex framework layers introduce substantial redundancy when paired with strong models. By exposing the low return on investment associated with over-engineered agent frameworks, this work provides empirical evidence supporting a "model capability over framework complexity" principle and advocates for a minimalist architectural paradigm in autonomous ML engineering.

Abstract

Recent autonomous machine learning engineering (MLE) agents have made significant progress on public leaderboards. Often motivated by progress stagnation over long-horizon cycles and limited Large Language Model (LLM) primitives, modern MLE agents are deployed on top of increasingly elaborate machinery: multi-agent orchestrators, dedicated retrieval subagents, and more. While such harnesses expand, the use of more primitive but improved coding agents - where LLMs have direct access to the execution environment through read, write, and bash primitives - has received little attention in the field. In this paper we find that, under an equal time budget and the same frontier LLM backbone, open-source state-of-the-art harnesses provide no advantages over a single session of a minimal-harness coding agent baseline, pointing to the backbone as the primary driver for performance. Via a series of large-scale systematic ablation studies, we argue that the machinery layers become redundant in the coding agent setting. We conclude that the effort spent elaborating hand-crafted harnesses around strong models yields poor returns for current MLE benchmarks.

Problem

• Autonomous Machine Learning Engineering
• Agent Harness
• Large Language Models
• Coding Agents
• Ablation Studies

Innovation

• Autonomous Machine Learning Engineering
• Coding Agents
• Minimal Harness
• Ablation Studies
• Large Language Models