State2State: Environment-Derived Mid-Training for LLM Agents

2026-08-05

AI Summary

This work addresses the limitations of conventional large language model (LLM) agent training, which relies on manually designed tasks and supervisory signals, thereby suffering from poor scalability and limited diversity. The authors propose an environment-driven, self-guided intermediate training paradigm in which agents autonomously explore their surroundings and automatically convert goal states into reachability tasks—eliminating the need for expert demonstrations or handcrafted task specifications. This approach pioneers a fully environment-interaction-based framework for generating verifiable training objectives, leveraging a state-to-state (State2State) mapping mechanism combined with rule-based verification and reinforcement learning policy initialization. Evaluated in ALFWorld and ScienceWorld, the method enables unsupervised agent training and significantly enhances downstream task performance and sample efficiency when used as an effective pre-training phase.

Abstract

Training LLM agents commonly relies on supervised fine-tuning from expert trajectories or online reinforcement learning over human-specified tasks with handcrafted verifiers. Though effective, both remain bottlenecked by externally specified tasks and supervision signals, limiting the scalability and diversity of agent training. We study an environment learning paradigm in which agents acquire interaction and manipulation capabilities solely through environment interaction, without externally specified tasks. We propose State2State, an environment-derived mid-training method that converts explored environment states into training objectives, challenging agents to reach a specified target state. By deriving tasks from environment exploration and verifying success through rule-based state matching, State2State provides scalable and verifiable training objectives without expert supervision or manual task design. Experiments on ALFWorld and ScienceWorld show that State2State improves agent performance as a standalone environment-learning stage in most settings. As initialization for downstream RL, it further improves final performance and learning efficiency, with promising evidence of cross-environment generalization.

Problem

• LLM agents
• environment interaction
• task-free training
• scalable training
• generalization

Innovation

• State2State
• environment-derived training
• self-supervised agent learning
• state-based task generation
• LLM agents