imitation learning

This work addresses the high barrier to entry in existing imitation learning methods, which typically require extensive expert demonstrations and continuous monitoring, rendering them inaccessible to non-expert users. To overcome this limitation, we propose an interactive policy restructuring and training framework that enables ordinary users without machine learning expertise to dynamically adjust an agent’s policy structure and parameters through natural-language instructions and a minimal number of demonstrations. Our approach integrates instruction parsing, dynamic policy architecture evolution, online imitation learning, and human-in-the-loop interaction mechanisms. A user study (N=34) demonstrates that, compared to baseline methods, the proposed framework significantly enhances policy robustness and trustworthiness while maintaining high usability.

Conventional policy optimization relies heavily on approximate dynamic programming or reinforcement learning (RL) frameworks, requiring explicit modeling of actions, control signals, and reward functions—limiting generalizability across diverse control and learning tasks. Method: This paper introduces a novel paradigm that embeds parameterized policies directly into autonomous dynamical systems, enabling joint optimization of policy parameters and system dynamics at the continuous-time dynamical level—bypassing RL-specific abstractions. Contribution/Results: The approach unifies behavior cloning, mechanism design, system identification, and state estimation within a single differentiable optimization framework, without requiring reward engineering or action-space specification. Theoretically, its gradient updates are shown to be equivalent to standard policy gradients, natural gradients, and PPO updates. Empirically, it achieves performance on par with state-of-the-art RL methods across diverse control benchmarks and provides a more intrinsic, fully differentiable foundation for optimizing generative AI systems.

To address low sample efficiency, passive student exploration, and weak generalization caused by imperfect teacher demonstrations in policy distillation, this paper proposes Proximal Policy Distillation (PPD). PPD is the first to integrate Proximal Policy Optimization (PPO) into the policy distillation framework, unifying student-driven distillation with PPO’s self-feedback optimization mechanism—enabling active student exploration and leveraging auxiliary rewards collected during distillation. A KL-divergence constraint ensures behavioral consistency between teacher and student policies, while PPD supports multi-scale student networks (smaller, equal-sized, or larger than the teacher). Evaluated on ATARI, MuJoCo, and Procgen benchmarks, PPD consistently outperforms both student-distill and teacher-distill variants in both sample efficiency and final policy performance. Notably, it demonstrates superior robustness and generalization under suboptimal teacher demonstrations. The implementation is open-sourced as sb3-distill, built upon stable-baselines3.

This work addresses the challenge of simultaneously ensuring behavioral preference satisfaction and monotonic policy improvement while maintaining efficient exploration in policy optimization. We propose the ε-retraining framework, which introduces (i) an iterative retraining region construction mechanism that dynamically identifies preference-violating regions in the state space via behavior bias localization; (ii) a decaying ε-scheduling strategy to jointly balance global exploration and local correction; and (iii) neural network formal verification—using ReLU partitioning and linear programming—to quantify preference adherence. Evaluated across motion control, navigation, and power grid dispatch tasks with over one hundred random seeds, our method achieves a 37.2% increase in preference compliance rate and accelerates convergence by 2.1×, significantly improving sample efficiency and policy reliability.

Embodied agents struggle with understanding low-level physical dynamics, cross-task generalization, and zero-shot environmental adaptation—particularly in the absence of task-specific demonstrations or customized simulation environments. Method: This paper proposes a multi-role collaborative online embodied agent framework, introducing the novel “role division” paradigm. It integrates vision-language models (VLMs), embodied reasoning agents, real-time visuomotor closed-loop control, and a modular dialogue architecture to enable online semantic grounding of policies and dynamic recalibration of action distributions. Contribution/Results: The framework achieves cross-task, cross-environment, and cross-platform deployment without requiring new demonstrations, simulation retraining, or hardware adaptation. Evaluated on both simulated and real robotic platforms, it demonstrates significantly improved manipulation success rates, validating its strong robustness and zero-shot adaptability.

This work addresses the challenge that GUI agents in closed-loop execution often encounter out-of-distribution states—caused by policy drift from expert trajectories—leading to decision failures due to insufficient supervision. To mitigate this, the paper proposes an iterative self-improvement framework in which an unskilled-guided policy actively explores such deviated states, while a skill-guided policy generates successful trajectory continuations. These continuations, combined with the original expert data, provide comprehensive supervision. The key innovation lies in introducing, for the first time, a skill-guided continuation distillation mechanism that extracts critical skills—such as continuation plans and target subtasks—from both successful and failed trajectories to fill supervision gaps. Evaluated on the OSWorld-Verified benchmark, the method significantly boosts success rates of three base models from approximately 30% to over 50%, demonstrating its effectiveness and generalizability.

This work addresses the challenges of low sample efficiency and performance degradation commonly encountered when fine-tuning large behavioral models via conventional reinforcement learning in robotic dexterous manipulation tasks with sparse rewards. The authors propose a coherent off-policy improvement method that integrates inverse reinforcement learning with behavioral cloning: a dense reward function is learned from expert demonstrations, and the pretrained policy is efficiently optimized within a theoretically grounded imitation learning framework while preserving its initial optimality. Evaluated across six sparse-reward tasks, the approach consistently maintains or improves performance, achieving success rates exceeding 90% on five complex tasks and significantly outperforming sparse-reward reinforcement learning baselines.

This work addresses the high cost and suboptimality of acquiring high-quality demonstration data in imitation learning, particularly the lack of scalable data sources for goal-conditioned control tasks. To overcome this challenge, the authors propose an efficient data generation and augmentation framework that leverages trajectory optimization to automatically produce thousands of near-optimal trajectories within minutes on a standard laptop. By relabeling intermediate states along these trajectories as new goals, the training dataset is expanded by an order of magnitude. A lightweight goal-conditioned policy trained on this augmented dataset—containing fewer than 80,000 parameters—achieves near-optimal performance and high success rates across multiple tasks. Moreover, its inference speed exceeds that of the trajectory optimization solver by over 6,000×, substantially improving generalization and enabling practical deployment on embedded systems.

Acquiring expert demonstration data on real robots is prohibitively expensive, limiting the generalization and robustness of behavior cloning. To address this challenge, this work proposes ExpertGen, a framework that leverages imperfect priors—such as human or large language model demonstrations—in simulation to enable efficient and safe policy transfer. By freezing a pretrained diffusion policy and optimizing only its initial noise, ExpertGen integrates diffusion models, reinforcement learning, and DAgger to significantly improve task success rates under sparse rewards without requiring reward engineering, while preserving behavioral similarity to human demonstrations. Experimental results demonstrate that the method achieves success rates of 90.5% in industrial assembly tasks and 85% in long-horizon manipulation tasks, substantially outperforming baseline approaches, and successfully transfers to real-world robotic deployment.

This work addresses the inefficiency of existing reinforcement learning methods that often disregard available suboptimal baseline policies, resulting in high training costs and low task success rates. We propose a model-free policy augmentation framework that leverages a dynamic arbitration mechanism: during early training, control is delegated to a functional baseline policy to ensure goal reachability, and is gradually transferred to a learnable policy, ultimately yielding a high-performance policy independent of the baseline. We formally define functional baselines for the first time and integrate probabilistic reachability analysis to design the transfer mechanism, providing theoretical guarantees on the lower bound of goal achievement probability for the final policy. Experiments on continuous control benchmarks demonstrate that our method achieves competitive or superior returns compared to state-of-the-art approaches while consistently maintaining the highest goal success rate throughout both training and standalone deployment.