COMAP: Co-Evolving World Models and Agent Policies for LLM Agents

2026-06-01

AI Summary

Existing textual world models are static after training, struggling to adapt to state-action distribution shifts induced by evolving agent policies, and their reliance on external rewards limits applicability in real-world settings. This work proposes the first co-evolutionary framework between a world model and an agent policy: a large language model constructs a textual world model that predicts future states for candidate actions; the agent employs a future-aware reflection mechanism to assess action reliability and refine its policy; and the resulting interaction trajectories continuously update the world model via self-distillation, forming a closed-loop reinforcement learning architecture that operates without external rewards. The method significantly outperforms baselines—e.g., achieving a 16.75% improvement over Qwen3-4B—on benchmarks spanning embodied task planning, web navigation, and tool use, while demonstrating continual improvement in world model prediction accuracy through interaction.

Abstract

Equipping language agents with world models enables them to anticipate environment dynamics and evaluate candidate actions before execution. However, existing textual world models are typically fixed after training, preventing them from adapting to the on-policy state-action distributions induced by an evolving agent. Meanwhile, agent-improvement methods often rely on external rewards or verifiers, limiting their applicability in realistic interactive environments. In this paper, we propose COMAP, a novel framework that co-evolves textual world models and agent policies through closed-loop interaction. At each decision step, the world model predicts future state feedback for candidate actions, and the agent performs future-aware reflection by estimating the reliability of this feedback and refining its action accordingly. The resulting on-policy trajectories are then used to update the world model via self-distillation, allowing it to better match the agent's evolving interaction distribution. Across embodied task planning, Web navigation, and tool-use benchmarks, COMAP consistently outperforms competitive baselines, e.g., +16.75% relative improvement with Qwen3-4B. Further analyses show that the co-evolutionary loop improves the world model's prediction accuracy over time and leads to more effective long-horizon decision-making. Our code is available at: https://github.com/loyiv/CoMAP.

Problem

• world models
• agent policies
• co-evolution
• on-policy adaptation
• language agents

Innovation

• co-evolution
• world model
• LLM agent
• self-distillation
• future-aware reflection