Seeking Consistent Flat Minima for Better Domain Generalization via Refining Loss Landscapes

2024-12-18

arXiv.org

AI Summary

Existing domain generalization methods overlook cross-domain consistency of loss landscapes when optimizing flat minima, hindering simultaneous convergence to optimal flat solutions across all source domains and limiting generalization performance. This paper proposes the Self-Feedback Training (SFT) framework—an iterative training paradigm that introduces, for the first time, a cross-domain loss landscape consistency constraint. By dynamically detecting and correcting inter-domain landscape inconsistencies, SFT jointly optimizes multi-source domain loss surfaces to achieve cross-domain alignment of flat minima. The method integrates a landscape consistency metric, gradient-driven landscape smoothing regularization, and an extension of Sharpness-Aware Minimization (SAM). Evaluated on five DomainBed benchmarks, SFT with ResNet-50 and ViT-B/16 backbones achieves average improvements of 2.6% and 1.5% over SAM, respectively, demonstrating significantly enhanced out-of-distribution generalization capability.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Search and Optimization: Learning to Search
• Computer Vision: Learning & Optimization for CV

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Domain generalization aims to learn a model from multiple training domains and generalize it to unseen test domains. Recent theory has shown that seeking the deep models, whose parameters lie in the flat minima of the loss landscape, can significantly reduce the out-of-domain generalization error. However, existing methods often neglect the consistency of loss landscapes in different domains, resulting in models that are not simultaneously in the optimal flat minima in all domains, which limits their generalization ability. To address this issue, this paper proposes an iterative Self-Feedback Training (SFT) framework to seek consistent flat minima that are shared across different domains by progressively refining loss landscapes during training. It alternatively generates a feedback signal by measuring the inconsistency of loss landscapes in different domains and refines these loss landscapes for greater consistency using this feedback signal. Benefiting from the consistency of the flat minima within these refined loss landscapes, our SFT helps achieve better out-of-domain generalization. Extensive experiments on DomainBed demonstrate superior performances of SFT when compared to state-of-the-art sharpness-aware methods and other prevalent DG baselines. On average across five DG benchmarks, SFT surpasses the sharpness-aware minimization by 2.6% with ResNet-50 and 1.5% with ViT-B/16, respectively. The code will be available soon.

Problem

• Seeking consistent flat minima across domains
• Refining loss landscapes for better generalization
• Reducing inconsistency in multi-domain loss landscapes

Innovation

• Self-Feedback Training for consistent flat minima
• Refining loss landscapes across domains iteratively
• Measuring and reducing loss landscape inconsistency