Gradient-Guided Annealing for Domain Generalization

2025-02-27

AI Summary

In domain generalization (DG), gradient conflicts across multiple source domains often cause premature convergence to non-robust local minima, hindering learning of domain-invariant features. To address this, we propose Gradient-Guided Annealing (GGA), the first method to embed gradient direction alignment into the early-stage parameter annealing process. GGA explicitly mitigates inter-domain gradient conflicts via direction-aware gradient regularization and a dynamic annealing schedule, guiding optimization toward more robust domain-invariant minima. As a plug-and-play module, GGA consistently enhances existing DG methods: it achieves state-of-the-art or highly competitive performance on five mainstream image-based DG benchmarks, and when integrated with current approaches, yields average accuracy improvements of 3.2%–5.8%.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Adversarial Attacks & Robustness
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Domain Generalization (DG) research has gained considerable traction as of late, since the ability to generalize to unseen data distributions is a requirement that eludes even state-of-the-art training algorithms. In this paper we observe that the initial iterations of model training play a key role in domain generalization effectiveness, since the loss landscape may be significantly different across the training and test distributions, contrary to the case of i.i.d. data. Conflicts between gradients of the loss components of each domain lead the optimization procedure to undesirable local minima that do not capture the domain-invariant features of the target classes. We propose alleviating domain conflicts in model optimization, by iteratively annealing the parameters of a model in the early stages of training and searching for points where gradients align between domains. By discovering a set of parameter values where gradients are updated towards the same direction for each data distribution present in the training set, the proposed Gradient-Guided Annealing (GGA) algorithm encourages models to seek out minima that exhibit improved robustness against domain shifts. The efficacy of GGA is evaluated on five widely accepted and challenging image classification domain generalization benchmarks, where its use alone is able to establish highly competitive or even state-of-the-art performance. Moreover, when combined with previously proposed domain-generalization algorithms it is able to consistently improve their effectiveness by significant margins.

Problem

• Improving domain generalization effectiveness
• Aligning gradients across data distributions
• Seeking robust minima against domain shifts

Innovation

• Gradient-Guided Annealing
• domain conflict alleviation
• parameter value discovery