mixed-environment rl

This work addresses the limitations of traditional reinforcement learning (RL) environment families, which are typically handcrafted and thus labor-intensive, error-prone, and difficult to scale. To overcome these challenges, the authors propose a model-driven approach that integrates model-driven engineering with RL environment generation for the first time. Their method employs a hybrid genetic algorithm combining global population-based search with heuristic local search, and expresses variation and constraints through model transformations. By leveraging a model transformation engine, the approach automatically constructs diverse, constraint-compliant environment families, effectively supporting advanced training paradigms such as curriculum learning. Experimental evaluation in a wildfire mitigation scenario demonstrates the method’s efficacy, yielding significantly improved training efficiency for RL agents.

This study addresses the low exploration efficiency and substantial engineering overhead caused by uniform sampling in large-scale reinforcement learning by proposing a capability-boundary-oriented adaptive sampling mechanism. This method dynamically focuses on task configurations at the capability frontier within million-scale parallel simulation environments, overcoming ultra-large-scale exploration bottlenecks and significantly improving the utilization of learning signals. Furthermore, by integrating sim-to-real reinforcement learning with visual policy distillation techniques, this work achieves zero-shot transfer to physical robots for both legged locomotion over complex terrains and precision assembly tasks. These results effectively resolve the generalization challenges that remain difficult for conventional approaches.

Automated environment configuration remains a persistent challenge in software engineering, where existing large language models exhibit limited capability. This paper proposes a lightweight and efficient solution: an end-to-end framework for localized environment setup, built upon the Qwen3-8B model and integrating supervised fine-tuning with verifiable-reward reinforcement learning (RLVR). The approach significantly improves the correctness and task-specific adaptability of generated Bash scripts. Evaluated on the EnvBench-Python benchmark, it achieves performance comparable to Qwen3-32B and GPT-4o—marking the first instance enabling real-time execution on consumer-grade hardware. We publicly release the training code and model checkpoints, establishing a new paradigm for reproducible large-scale experimentation and low-resource environment configuration.

This work proposes the LLM-as-Environment-Engineer framework, which leverages a large language model (Qwen3-4B) as an “environment engineer” to dynamically optimize reinforcement learning training environments. Addressing the common reliance on manually tuned settings and the absence of automated, performance-driven environment adaptation, the framework analyzes policy failure trajectories, behavioral summaries, and environmental statistics to automatically reconstruct a multi-dimensional, configurable MAPF-FrozenLake environment. Experimental results demonstrate that reinforcement learning agents fine-tuned within this closed-loop, LLM-driven optimization process not only develop enhanced self-diagnostic capabilities to guide environmental refinements but also achieve superior overall performance—outperforming both larger closed-source models such as GPT and Gemini and fixed-environment baselines—thereby validating the efficacy and novelty of dynamic environment design in reinforcement learning.

To address the challenges of prohibitively long training times, high real-world deployment costs, and poor sim-to-real transfer performance in multi-agent reinforcement learning (MARL) for cyber-physical vehicular systems, this paper proposes a hybrid-reality digital twin framework. The framework introduces an on-demand dynamic parallelization-based load scheduling mechanism to enable elastic scaling of simulation resources, coupled with a systematic domain randomization strategy to enhance policy generalization. Our approach enables efficient collaborative MARL training and zero-shot sim-to-real transfer. Extensive evaluation across cooperative and adversarial traffic scenarios demonstrates its effectiveness: training time is reduced by up to 76.3%, while the sim-to-real performance gap is narrowed to just 2.9%. The method thus achieves a compelling balance between computational scalability and physical deployability.

This study addresses a prevalent conflation in reinforcement learning research between two distinct objectives involving simulators: solving the simulator as an end in itself versus treating it as a proxy for a real-world deployment environment. The former seeks high returns within the simulated domain, while the latter aims to transfer learned policies to the physical world. These goals entail fundamentally different algorithmic constraints, methodological requirements, and evaluation criteria. Through conceptual clarification, illustrative case studies, and controlled experiments, this work exposes the pitfalls arising from conflating these roles, delineates their respective appropriate use cases, and calls upon the research community to align experimental design, evaluation metrics, and algorithm development with the intended purpose of simulator usage.

This work addresses the high computational cost of high-fidelity simulation models, which hinders efficient training and retraining of reinforcement learning agents in dynamic environments. To overcome this limitation, the paper proposes a learnable surrogate modeling framework tailored for dynamic settings, which approximates the input–output mapping of high-fidelity simulations to substantially reduce training overhead when system dynamics, parameters, or reward structures change. By integrating discrete-event simulation, reinforcement learning, and data-driven surrogate modeling techniques, the framework enables rapid adaptation of policies to environmental shifts. Empirical evaluation in stochastic service systems demonstrates significant acceleration in both initial training and retraining processes, thereby enhancing the adaptability of reinforcement learning policies to evolving conditions.

Current agent training lacks realistic synthetic environments that are executable, resettable, and exhibit behavioral depth—particularly in login-restricted and state-dependent scenarios. This work proposes a framework for specifying environments that compile into stateful applications, enabling co-evolutionary training wherein evaluation trajectories simultaneously inform environment refinement and agent optimization. The framework integrates a database-driven task scorer, dense step-level rewards, and a grounding validation mechanism. Experiments demonstrate that a 9B-parameter model improves from 36.5% to 67.1% average performance across 14 benchmarks, approaching the capabilities of much larger state-of-the-art models. A single round of environment repair doubles performance, and under reinforcement learning settings, the agent achieves 68.0% success, substantially enhancing online accuracy.

Current agents exhibit limited generalization when confronted with out-of-distribution environmental changes, such as shifts in interaction rules, dynamics, or observation feedback. This work proposes “environment expansion”—enhancing cross-environment generalization by broadening the distribution of executable rule sets an agent interacts with, rather than merely increasing the number of trajectories or tasks. We formally distinguish trajectory expansion, task expansion, and environment expansion, establishing a unified taxonomy and highlighting that distributional expansion at the environment level is essential for robust, general-purpose agents. Scalable environments are constructed via two paradigms: procedural generators and generative world models, integrated with state-aware learning mechanisms to enable cross-environment adaptation. This study provides a theoretical framework and technical pathway toward measurable and controllable general agents, significantly improving their adaptability and robustness in unseen environments.

为解决长周期任务中奖励稀疏问题，提出通过构建反馈丰富的环境(FEEs)来增强自主代理训练效果，实验显示该方法能稳定训练并促进状态空间探索。