All you need is log

2026-06-25

AI Summary

This study addresses the open problem of constructing a unified family of divergences for multi-distribution comparison that simultaneously satisfies data-processing monotonicity and product additivity. By employing a structural axiomatic approach, we fully characterize all such functionals and prove they must take the form of positive integrals of a novel multivariate coincidence divergence. We introduce this multivariate coincidence divergence together with its four-parameter structure, which subsumes the classical Rényi divergence as a special case. The universality of the proposed divergence is independently corroborated through five distinct perspectives: information geometry, functional equations, Kolmogorov–Nagumo averages, error exponents in hypothesis testing, and a game-theoretic interpretation. Theoretical analysis establishes the uniqueness and completeness of this divergence family, and its efficacy is demonstrated via a concrete example with three distributions (W=3), numerical experiments, and conditional extensions.

Abstract

Comparing two probability distributions is a basic building block of statistics and machine learning, and the right family is well understood: the Rényi divergences of order $α\in[0,\infty]$ are the unique family monotone under data processing and additive on independent products. Many problems instead compare more than two distributions at once -- multi-population fairness, multi-prior PAC-Bayes bounds, multi-hypothesis testing -- and the right multi-distribution generalization of the Rényi family has been an open question. We characterize it. Every functional of $W$-tuples of distributions that is monotone under data processing and additive on independent products is a positive integral of multi-way coincidence divergences $C_α(π_1,\dots,π_W) := -\log\int π_1^{α_1}\cdotsπ_W^{α_W}$ (with $\sum_k α_k = 1$) over a parameter space with four strata: the simplex interior; mixed-sign exponent cones (the analogue of Rényi orders $>1$); a tropical boundary at infinity carrying max-divergences; and pairwise Kullback-Leibler edges at the simplex vertices. Each stratum is necessary -- the destination of an explicit data-processing-monotone, product-additive divergence the others cannot reproduce -- and each is a clean limit of simplex-interior atoms. The same family arises from five independent routes -- the structural axioms, Kolmogorov-Nagumo means with Rényi's entropy axiomatics, classical entropy characterizations, multi-hypothesis testing error exponents, and a multi-lottery betting interpretation -- structural evidence that this is the canonical multi-distribution Rényi calculus rather than an artefact of any one axiomatic input. The two-prior case recovers the standard Rényi result; a worked $W=3$ instance, numerical verification, and a conditional extension round out the treatment.

Problem

• Rényi divergence
• multi-distribution comparison
• data processing monotonicity
• product additivity
• multi-hypothesis testing

Innovation

• multi-distribution divergence
• Rényi divergence
• data processing monotonicity
• multi-way coincidence divergence
• product additivity