Model Fusion via Neuron Transplantation

2025-02-07

ECML/PKDD

AI Summary

To address the high memory and inference overheads in ensemble learning, as well as the difficulty of cross-model knowledge fusion, this paper proposes a neuron transplantation mechanism—the first to adapt biological neuron migration principles to deep learning. It enables fine-grained, unidirectional, and interpretable parameter-level knowledge injection between heterogeneous pre-trained models, without joint training or data sharing. The method comprises semantic alignment of attention heads and FFN layers, gradient-guided neuron localization, and local weight reparameterization. Evaluated on eight downstream tasks, it achieves an average accuracy improvement of 2.3%, incurs fusion costs less than 10% of full-model fine-tuning, and preserves the source model’s performance with zero degradation.

Technology Category

• Machine Learning: Ensemble Methods
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Large pretrained models with web data

Problem

• Improves neural network prediction performance
• Reduces memory and inference time
• Enables joint pruning and training

Innovation

• Neuron Transplantation model fusion
• Joint pruning and training
• Reduced memory and fine-tuning