Domain Feature Collapse: Implications for Out-of-Distribution Detection and Solutions

2025-12-03

AI Summary

Single-domain training causes catastrophic failure in out-of-distribution (OOD) detection due to domain-feature collapse induced by supervised learning—models implicitly discard domain information, i.e., mutual information (I(x_d; z) o 0), rendering them incapable of distinguishing cross-domain OOD samples. Method: We first establish, from an information-theoretic perspective, the inevitability of this collapse using the information bottleneck principle and Fano’s inequality. We then propose a “domain filtering” mechanism that explicitly preserves domain-relevant representations during inference, guiding pre-trained model fine-tuning or layer freezing strategies. Contribution/Results: We introduce Domain Bench, a dedicated benchmark for evaluating domain-aware OOD detection. Experiments on MNIST and other datasets demonstrate substantial improvements in cross-domain OOD detection performance. Empirically, domain filtering restores (I(x_d; z) > 0), fundamentally mitigating generalization failure caused by single-domain training.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Application Domains: Misinformation & Fake News
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Why do state-of-the-art OOD detection methods exhibit catastrophic failure when models are trained on single-domain datasets? We provide the first theoretical explanation for this phenomenon through the lens of information theory. We prove that supervised learning on single-domain data inevitably produces domain feature collapse -- representations where I(x_d; z) = 0, meaning domain-specific information is completely discarded. This is a fundamental consequence of information bottleneck optimization: models trained on single domains (e.g., medical images) learn to rely solely on class-specific features while discarding domain features, leading to catastrophic failure when detecting out-of-domain samples (e.g., achieving only 53% FPR@95 on MNIST). We extend our analysis using Fano's inequality to quantify partial collapse in practical scenarios. To validate our theory, we introduce Domain Bench, a benchmark of single-domain datasets, and demonstrate that preserving I(x_d; z)>0 through domain filtering (using pretrained representations) resolves the failure mode. While domain filtering itself is conceptually straightforward, its effectiveness provides strong empirical evidence for our information-theoretic framework. Our work explains a puzzling empirical phenomenon, reveals fundamental limitations of supervised learning in narrow domains, and has broader implications for transfer learning and when to fine-tune versus freeze pretrained models.

Problem

• Explains catastrophic OOD detection failure in single-domain training
• Proves domain feature collapse via information bottleneck theory
• Introduces Domain Bench and domain filtering as a solution

Innovation

• Introduces Domain Bench benchmark for single-domain datasets
• Uses domain filtering to preserve domain-specific information
• Applies Fano's inequality to quantify partial feature collapse