Policy Plasticity Matters in Offline-to-Online Reinforcement Learning: Refitting Offline Policies for Online Adaptation

2026-09-27

AI Summary

This study addresses the degradation of network plasticity in offline reinforcement learning policies during online adaptation, which substantially impedes effective fine-tuning. To overcome this challenge, we propose REFIT, a novel framework that reconceptualizes the offline-to-online transition from a network plasticity perspective. Specifically, REFIT transfers the pretrained offline policy into a randomly initialized student network via knowledge distillation and incorporates stochastic unit freezing to enable lightweight policy refitting, thereby effectively restoring model plasticity. Extensive evaluations on the D4RL and OGBench benchmarks demonstrate that our approach significantly outperforms existing plug-and-play methods, empirically validating the critical role of plasticity restoration in facilitating online adaptation.

Abstract

Offline-to-Online Reinforcement Learning (O2O RL) has emerged as a practical paradigm that pre-trains the policy using static offline datasets and subsequently adapts the policy through online interactions. Existing O2O methods primarily address the transition through value calibration, while generally treating the offline-trained policy as a given initialization. We instead study O2O adaptation from the perspective of network plasticity, asking whether the offline-trained policy remains sufficiently adaptable for online learning. Controlled experiments show that prolonged optimization on static offline data progressively reduces network plasticity even after offline performance has largely saturated, and that lower plasticity is associated with weaker subsequent online improvement. Motivated by these observations, we propose REstoring plasticity via Fresh Initialization and policy Transfer (REFIT), a lightweight model-level method for the O2O transition. Before online fine-tuning, REFIT distills the offline policy into a freshly initialized student while temporarily freezing a random subset of student units, transferring the learned offline behavior to a more plastic policy initialization. Extensive experiments on D4RL and OGBench demonstrate that REFIT consistently achieves higher aggregate performance than existing O2O plug-in methods across both Cal-QL and IQL backbones, while plasticity diagnostics and ablations provide further evidence of restored network plasticity.

Problem

• Offline-to-Online Reinforcement Learning
• Network Plasticity
• Policy Adaptation
• Fine-tuning

Innovation

• Offline-to-Online Reinforcement Learning
• Network Plasticity
• Policy Distillation
• Fresh Initialization
• REFIT