In-the-Flow Agentic System Optimization for Effective Planning and Tool Use

2025-10-07

AI Summary

Existing tool-augmented LLM reinforcement learning methods rely on a single static policy, exhibiting poor generalization and limited scalability in long-horizon, multi-tool tasks; meanwhile, mainstream agent systems struggle to balance trainability with real-time online optimization. This paper proposes AgentFlow—the first trainable agent framework enabling online policy optimization across multi-turn interactions—leveraging coordinated planning, execution, verification, and generation modules alongside dynamically evolving memory to convert long-horizon sparse rewards into trajectory-level single-step updates. We introduce Flow-GRPO, a novel algorithm integrating grouped refinement policy optimization, normalized advantage estimation, and flow-aware credit assignment. Evaluated on ten benchmarks, AgentFlow achieves an average 14.9% improvement in search accuracy using only a 7B model—surpassing GPT-4o—and significantly enhances complex task planning capability and tool invocation reliability.

Technology Category

• Multiagent Systems: Multiagent Learning
• Search and Optimization: Learning to Search
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Outcome-driven reinforcement learning has advanced reasoning in large language models (LLMs), but prevailing tool-augmented approaches train a single, monolithic policy that interleaves thoughts and tool calls under full context; this scales poorly with long horizons and diverse tools and generalizes weakly to new scenarios. Agentic systems offer a promising alternative by decomposing work across specialized modules, yet most remain training-free or rely on offline training decoupled from the live dynamics of multi-turn interaction. We introduce AgentFlow, a trainable, in-the-flow agentic framework that coordinates four modules (planner, executor, verifier, generator) through an evolving memory and directly optimizes its planner inside the multi-turn loop. To train on-policy in live environments, we propose Flow-based Group Refined Policy Optimization (Flow-GRPO), which tackles long-horizon, sparse-reward credit assignment by converting multi-turn optimization into a sequence of tractable single-turn policy updates. It broadcasts a single, verifiable trajectory-level outcome to every turn to align local planner decisions with global success and stabilizes learning with group-normalized advantages. Across ten benchmarks, AgentFlow with a 7B-scale backbone outperforms top-performing baselines with average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks, even surpassing larger proprietary models like GPT-4o. Further analyses confirm the benefits of in-the-flow optimization, showing improved planning, enhanced tool-calling reliability, and positive scaling with model size and reasoning turns.

Problem

• Optimizing agentic systems for long-horizon planning with diverse tools
• Enhancing multi-turn interaction through trainable in-the-flow framework
• Improving sparse-reward credit assignment in reinforcement learning for reasoning

Innovation

• Trainable agentic framework with four specialized modules
• Flow-GRPO enables multi-turn policy optimization with outcome alignment
• Evolving memory coordinates planner decisions for long-horizon tasks