Restoring Neural Network Plasticity for Faster Transfer Learning

2026-03-21

AI Summary

This work addresses the degradation of neural plasticity in pretrained models during transfer to downstream tasks, a phenomenon often caused by weight saturation that impairs adaptation to atypical data. To mitigate this issue, the study introduces— for the first time—a systematic neuroplasticity restoration mechanism in transfer learning through a lightweight, architecture-agnostic targeted weight reinitialization strategy applied prior to fine-tuning. The proposed method effectively alleviates weight saturation without altering the standard training pipeline and is compatible with both convolutional neural networks (CNNs) and Vision Transformers (ViTs). Extensive experiments demonstrate that this approach consistently accelerates convergence and improves final accuracy across multiple image classification benchmarks, all while incurring negligible computational overhead.

Technology Category

Machine Learning: Transfer, Domain Adaptation, Multi-Task LearningComputer Vision: Diffusion Models for VisionNatural Language Processing: Safety and Robustness

Application Category

Web Mining and Content Analysis: Large pretrained models with web dataSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingUser Modeling, Personalization and Recommendation: User modeling for targeted and personalized online advertising

Abstract

Transfer learning with models pretrained on ImageNet has become a standard practice in computer vision. Transfer learning refers to fine-tuning pretrained weights of a neural network on a downstream task, typically unrelated to ImageNet. However, pretrained weights can become saturated and may yield insignificant gradients, failing to adapt to the downstream task. This hinders the ability of the model to train effectively, and is commonly referred to as loss of neural plasticity. Loss of plasticity may prevent the model from fully adapting to the target domain, especially when the downstream dataset is atypical in nature. While this issue has been widely explored in continual learning, it remains relatively understudied in the context of transfer learning. In this work, we propose the use of a targeted weight re-initialization strategy to restore neural plasticity prior to fine-tuning. Our experiments show that both convolutional neural networks (CNNs) and vision transformers (ViTs) benefit from this approach, yielding higher test accuracy with faster convergence on several image classification benchmarks. Our method introduces negligible computational overhead and is compatible with common transfer learning pipelines.

Problem

• transfer learning
• neural plasticity
• pretrained models
• weight saturation
• fine-tuning

Innovation

• neural plasticity
• transfer learning
• weight re-initialization
• fine-tuning
• vision transformers