An Agency-Transferring Model-Free Policy Enhancement Technique

2026-06-08

AI Summary

This work addresses the inefficiency of existing reinforcement learning methods that often disregard available suboptimal baseline policies, resulting in high training costs and low task success rates. We propose a model-free policy augmentation framework that leverages a dynamic arbitration mechanism: during early training, control is delegated to a functional baseline policy to ensure goal reachability, and is gradually transferred to a learnable policy, ultimately yielding a high-performance policy independent of the baseline. We formally define functional baselines for the first time and integrate probabilistic reachability analysis to design the transfer mechanism, providing theoretical guarantees on the lower bound of goal achievement probability for the final policy. Experiments on continuous control benchmarks demonstrate that our method achieves competitive or superior returns compared to state-of-the-art approaches while consistently maintaining the highest goal success rate throughout both training and standalone deployment.

Abstract

Training reinforcement learning (RL) policies from scratch is costly: it requires careful reward and environment design, extensive tuning, and substantial computation. Yet many control problems already have a functional but suboptimal policy available as a baseline. This paper proposes a method for embedding such a baseline into the RL training process, simultaneously improving training efficiency relative to from-scratch methods and producing a learning policy that outperforms the baseline. At each step, the method arbitrates between the baseline policy and a trainable learning policy, initially relying strongly on the baseline policy and then progressively transferring agency to the learning policy. By the end of training, the learning policy is a standalone neural network that operates without baseline policy support. The paper formalizes what it means for the baseline policy to be functional: under this policy, the agent reaches a goal set and remains there with high probability. The proposed arbitration mechanism is designed to exploit this property during training, yielding high goal-reaching rates right from the beginning of training. A theoretical analysis provides a formal interpretation of this behavior under stated assumptions and extends it to the final baseline-free regime, where explicit lower bounds are derived for the goal-reaching probability of the standalone learning policy. Empirical results on continuous-control benchmarks show that the proposed method achieves returns that match or exceed those of competitive approaches, while maintaining the highest goal-reaching rates throughout training among the compared methods -- including in the final stage, where the learning policy operates without any baseline support.

Problem

• reinforcement learning
• baseline policy
• policy enhancement
• training efficiency
• goal-reaching

Innovation

• agency transfer
• model-free reinforcement learning
• baseline policy embedding
• goal-reaching guarantee
• policy arbitration