Matryoshka Agent: Unfolding Sub-Agents for Long-Horizon Machine Learning Engineering

2026-07-27

AI Summary

This work addresses the challenges of complex decision-making and iterative optimization in machine learning engineering, where long development cycles, high costs, and feedback-driven environments prevail. To tackle these issues, the authors propose a nested hierarchical agent framework: a high-level Orchestrator performs strategic planning, while low-level Sub-Agents execute tasks via standardized tool interfaces, thereby decoupling exploration from execution. This architecture alleviates the computational burden of long-context reasoning and enables efficient iteration and scalable training. Experimental results demonstrate substantial performance gains across diverse machine learning engineering tasks—Qwen3-4B-Instruct achieves performance comparable to o4-mini, and Qwen3-30B-Coder shows a relative improvement of up to 36.7%.

Abstract

Machine learning engineering (MLE) tasks require long-horizon decision making over iterative solution debugging and refinement, under expensive and feedback-driven environment interactions. Developing and training a monolithic agent for such tasks is fundamentally challenging, as it must simultaneously manage extremely long and noisy contexts, explore vast solution spaces, and remain effective under limited model capacity and computational budgets. To address these challenges, we propose Matryoshka Agent, a unified hierarchical agent framework for complex long-horizon tasks. Matryoshka Agent decomposes agentic problem solving into a coordinated hierarchy of decision making and execution: a high-level Orchestrator maintains compact, long-horizon exploration states and issues strategic instructions, while lower-level Sub-Agents execute concrete solution attempts through direct environment interaction, mediated by standardized Tool interface. This design decouples strategic exploration from costly execution, substantially reducing the burden of long-context reasoning and enabling efficient iterative refinement. We further develop an efficient training paradigm for Matryoshka Agent. Experimental results on a broad range of MLE tasks with diverse model types and scales demonstrate that Matryoshka Agent is an effective and scalable paradigm for long-horizon MLE tasks and complex agentic problem solving. Notably, Matryoshka Agent enables Qwen3-4B-Instruct to reach Orchestrator performance comparable to o4-mini. Applying Matryoshka Agent to Qwen3-30B-Coder results in at most 36.7% relative performance gain.

Problem

• long-horizon decision making
• machine learning engineering
• agentic problem solving
• iterative refinement
• limited computational budget

Innovation

• Matryoshka Agent
• hierarchical agent
• long-horizon decision making
• sub-agent decomposition
• machine learning engineering