Asymptotically minimax regret by Bayes mixtures

1998-08-16

Proceedings. 1998 IEEE International Symposium on Information Theory (Cat. No.98CH36252)

AI Summary

This paper investigates the regret (redundancy) of sequential prediction, data compression, and gambling relative to smooth parametric families—specifically exponential families, general smooth parametric models, and Markov sources. It analyzes the asymptotic regret performance of Bayesian mixture distributions, particularly Jeffreys-prior-based variants, under maximum-likelihood estimation. The key contribution is the first rigorous proof that such Jeffreys-type mixture priors achieve asymptotically minimax regret over all these model classes, with redundancy converging at the optimal rate of $O(1/n)$. This rate matches the information-theoretic lower bound dictated by the Shtarkov normalized constant. By unifying tools from information geometry, asymptotic statistics, and normalized maximum-likelihood theory, the work establishes a fundamental connection between Bayesian mixtures and information-theoretic optimality. It thus provides a unified, theoretically grounded guarantee of optimality for universal coding and prediction.

Technology Category

• Machine Learning: Information Theory
• Reasoning under Uncertainty: Graphical Models
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks

Abstract

We study the problem of data compression, gambling and prediction of a sequence x/sup n/ = x/sub 1/x/sub 2/...x/sub n/ from a certain alphabet X, in terms of regret (Shtarkov 1988) and redundancy with respect to a general exponential family, a general smooth family, and also Markov sources. In particular, we show that variants of Jeffreys mixture asymptotically achieve their minimax values.

Problem

• Evaluating regret of Bayes mixtures vs maximum likelihood
• Achieving asymptotically minimax regret with Jeffreys prior variants
• Extending minimax regret to non-exponential families via modifications

Innovation

• Bayes mixtures achieve minimax regret
• Modified Jeffreys prior for non-exponential families
• Local exponential tilting enlarges family