Information from coincidences

2026-06-23

AI Summary

This work addresses the lack of a unified algebraic foundation for diverse variational formulations in information theory by introducing an algebraic mixture coincidence identity that integrates empirical distribution concentration, hypothesis testing error exponents, measure-change inequalities, and rare-pattern coincidence laws into a single coherent framework. Methodologically, it extends the variational characterizations of Rényi entropy and divergences to the W-prior simplex, accommodating unnormalized and continuously indexed priors, and incorporates an explicit “coincidence reward” term. The approach synergistically combines exponential family normalization, KL barycenter optimization, PAC-Bayes analysis, and large-alphabet sequence modeling. The resulting framework successfully recovers contrastive decoding mechanisms, effectively discriminates between relevant and diverse prior families in human genomic regulatory sequence analysis, and yields the asymptotic MAP error exponent for W-ary hypothesis testing.

Abstract

We prove a single algebraic mixed coincidence identity that unifies a broad swath of information-theoretic variational results. For any family of priors $\{π_i\}$ and real exponents $\{ α_i \}$, the log of the mixed count $E_{x\simν}\!\left[\prod_{i=1}^W π_i^{α_i}(x)\right]$ is simultaneously a Boltzmann coincidence weight, an exponential-family normalizer, a maximum-entropy value, and a KL-barycenter optimum. The identity yields a unified derivation of classical cornerstones of information theory: concentration of empirical distributions (Sanov-type decompositions and Gibbs conditioning), hypothesis-testing error exponents (Chernoff information and its multi-way analogue), change-of-measure inequalities (Donsker-Varadhan and PAC-Bayes), and laws governing rare-pattern coincidences (Erdos-Renyi run-length, iterative guesswork, rate-distortion, and birthday thresholds). Each is recovered as a specialization of the same algebraic equality. It strictly generalizes the classical Renyi entropy and divergence variational formulas (one and two priors respectively) to a $W$-prior simplex, and holds for unnormalized and continuum-indexed priors. Among its consequences are an exact multi-prior PAC-Bayes penalty that subtracts an explicit "coincidence bonus" from the usual single-prior posterior penalty, and the asymptotic MAP error exponent for $W$-ary hypothesis testing as an edge-restricted simplex optimum. We demonstrate the calculus at scale on two large alphabets encoding richly modeled sequential languages: on language-model next-token predictives where we recover contrastive decoding, and on human genomic regulatory sequence where it separates correlated from diverse prior families along a sliding-window trace.

Problem

• information theory
• variational formulas
• coincidence identity
• hypothesis testing
• PAC-Bayes

Innovation

• mixed coincidence identity
• variational information theory
• multi-prior PAC-Bayes
• Rényi generalization
• W-ary hypothesis testing