ReLU integral probability metric and its applications

2025-04-26

AI Summary

This paper addresses the challenge of measuring discrepancies between high-dimensional distributions. We propose a parameterized Integral Probability Metric (IPM) based on a single-node ReLU neural network. Our method is the first to employ a shallow ReLU network as the discriminator, yielding an IPM formulation that is analytically tractable, optimization-friendly, and endowed with theoretical convergence guarantees—significantly reducing sensitivity to hyperparameters. Compared to conventional nonparametric IPMs, our approach is computationally efficient, implementationally simple, and inherently supports covariate balancing and fair representation learning, facilitating causal and fairness-aware tasks. Extensive experiments across multiple benchmarks demonstrate that the proposed IPM matches or surpasses state-of-the-art methods in distribution discrimination, generalization, and robustness, while achieving the optimal theoretical convergence rate.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Relational Probabilistic Models
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

We propose a parametric integral probability metric (IPM) to measure the discrepancy between two probability measures. The proposed IPM leverages a specific parametric family of discriminators, such as single-node neural networks with ReLU activation, to effectively distinguish between distributions, making it applicable in high-dimensional settings. By optimizing over the parameters of the chosen discriminator class, the proposed IPM demonstrates that its estimators have good convergence rates and can serve as a surrogate for other IPMs that use smooth nonparametric discriminator classes. We present an efficient algorithm for practical computation, offering a simple implementation and requiring fewer hyperparameters. Furthermore, we explore its applications in various tasks, such as covariate balancing for causal inference and fair representation learning. Across such diverse applications, we demonstrate that the proposed IPM provides strong theoretical guarantees, and empirical experiments show that it achieves comparable or even superior performance to other methods.

Problem

• Measure discrepancy between probability distributions using parametric IPM
• Optimize discriminator parameters for good convergence and high-dimensional use
• Apply IPM in causal inference and fair representation learning

Innovation

• ReLU-based parametric IPM for distribution discrepancy
• Efficient algorithm with simple implementation
• Strong theoretical guarantees in diverse applications