SAFE-Merge: Data-Free Continual Model Merging with General Knowledge Preservation

2026-08-02

AI Summary

This work addresses the challenge of continually merging specialized models in data-free settings while preserving both pre-trained general knowledge and performance on previously learned tasks. The authors propose SAFE-Merge, a novel framework that explicitly prioritizes the safety of general knowledge during model fusion. It employs a risk-aware sparse masking strategy to selectively update only those parameters deemed safe for general knowledge retention and introduces a low-rank recovery mechanism to compensate for task-specific information loss in masked regions. Notably, this approach enables collaborative optimization without altering the original model parameters. Evaluated on vision and language benchmarks, SAFE-Merge achieves state-of-the-art H-scores and significantly outperforms NUFILT on long-sequence CLIP tasks while attaining the highest accuracy.

Abstract

Data-free continual model merging must incorporate a stream of specialized models while retaining both pretrained general knowledge and previously acquired tasks, without access to task data. Existing methods mainly merge task updates by suppressing interference among downstream tasks; while this protects previously acquired tasks, it overlooks the safety of the pretrained knowledge itself, whose erosion degrades generalization to held-out distributions and weakens the foundation for future task acquisition. We propose SAFE-Merge, a simple data-free continual-merging framework that first decides which parameter updates are safe to retain, and then recovers the task information lost through masking. Specifically, to ensure safety, risk-aware sparse masking selects parameter updates that carry task-specific information while posing low risk to general knowledge. Masked low-rank recovery then compensates for the lost task information using only the same retained parameter updates, while leaving all masked-out parameters strictly unchanged. Finally, the combined update is fused into the backbone, incurring no additional inference cost. Across vision and language benchmarks, SAFE-Merge consistently achieves the best H-score. On longer CLIP task sequences, it substantially improves H-score over NUFILT while also achieving the highest accuracy.

Problem

• data-free continual learning
• model merging
• general knowledge preservation
• task interference
• parameter update safety

Innovation

• data-free continual learning
• model merging
• general knowledge preservation
• risk-aware masking
• low-rank recovery