BD-Merging: Bias-Aware Dynamic Model Merging with Evidence-Guided Contrastive Learning

2026-03-04

AI Summary

This work addresses the susceptibility of model ensembles to biased and unreliable predictions under test-time distribution shifts. To this end, we propose an unsupervised, bias-aware dynamic fusion framework that jointly models cross-task semantic dependencies through an evidence head, quantifies inter-sample evidence consistency via an adjacency discrepancy score, and integrates discrepancy-aware contrastive learning with an unsupervised routing mechanism. This design enables adaptive and robust ensemble fusion in the presence of distributional shifts. Experimental results demonstrate that our method consistently outperforms existing fusion strategies across diverse tasks, significantly enhancing robustness under distribution shift while maintaining predictive accuracy.

Technology Category

• Machine Learning: Ensemble Methods
• Reasoning under Uncertainty: Relational Probabilistic Models
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Model Merging (MM) has emerged as a scalable paradigm for multi-task learning (MTL), enabling multiple task-specific models to be integrated without revisiting the original training data. Despite recent progress, the reliability of MM under test-time distribution shift remains insufficiently understood. Most existing MM methods typically assume that test data are clean and distributionally aligned with both the training and auxiliary sources. However, this assumption rarely holds in practice, often resulting in biased predictions with degraded generalization. To address this issue, we present BD-Merging, a bias-aware unsupervised model merging framework that explicitly models uncertainty to achieve adaptive reliability under distribution shift. First, BD-Merging introduces a joint evidential head that learns uncertainty over a unified label space, capturing cross-task semantic dependencies in MM. Second, building upon this evidential foundation, we propose an Adjacency Discrepancy Score (ADS) that quantifies evidential alignment among neighboring samples. Third, guided by ADS, a discrepancy-aware contrastive learning mechanism refines the merged representation by aligning consistent samples and separating conflicting ones. Combined with general unsupervised learning, this process trains a debiased router that adaptively allocates task-specific or layer-specific weights on a per-sample basis, effectively mitigating the adverse effects of distribution shift. Extensive experiments across diverse tasks demonstrate that BD-Merging achieves superior effectiveness and robustness compared to state-of-the-art MM baselines.

Problem

• Model Merging
• Distribution Shift
• Bias
• Multi-task Learning
• Uncertainty

Innovation

• Model Merging
• Distribution Shift
• Evidential Learning
• Contrastive Learning
• Uncertainty Modeling