On the Cardinality of Optimal Representations in the Binary-Source Information Bottleneck

2026-10-05

AI Summary

This study addresses the long-standing limitation in the Information Bottleneck (IB) framework, where the cardinality bound for optimal representations of binary sources has been constrained by generic upper bounds, thereby hindering computational efficiency. By exploiting the structural properties specific to the binary case, this work employs a separating hyperplane argument combined with concavity analysis of the ratio of second derivatives of entropy functions to transcend traditional generic bounds. It rigorously proves that the optimal representation for a binary source is itself binary, tightening the classical cardinality bound to the exact limit |U|≤|X|. This contribution not only establishes a theoretically optimal bound but also substantially reduces the computational complexity of solving IB problems.

Abstract

The information bottleneck (IB) seeks a representation $U$ of a source $X$ that retains as much information as possible about a target $Y$, subject to a constraint on $I(U;X)$. A classical argument shows that it suffices to consider representations with at most $|\mathcal{X}|+1$ symbols, and this bound is known to be tight whenever $|\mathcal{X}| \geq 3$. We show that the binary case behaves differently: if $X$ is binary and $Y$ is finite, then for every joint distribution of $(X,Y)$ and every rate constraint, the IB optimum is attained by a binary $U$. Hence the bound $|\mathcal{U}| \leq |\mathcal{X}|+1$ sharpens to $|\mathcal{U}| \leq |\mathcal{X}|$ for binary sources. The proof combines a separating hyperplane argument with the observation that, for a binary source, the ratio of the second derivatives of the two entropy functions involved is concave.

Problem

• Information Bottleneck
• Binary Source
• Optimal Representation
• Cardinality

Innovation

• Information Bottleneck
• Binary Source
• Cardinality Bound
• Optimal Representation
• Separating Hyperplane