Rethinking the Information Bottleneck: Structured Decomposition under Label-Induced Partitions

2026-10-01

AI Summary

This study addresses the limitation of the standard Information Bottleneck (IB) in disentangling label-relevant structures from irrelevant noise, which renders models prone to overfitting in few-shot scenarios. Building upon a label-induced partitioned reconstruction IB, this work proposes a dual-bottleneck framework that independently regulates global capacity and intra-conditional information. By achieving an exact decomposition of the conditional KL divergence and introducing a simplex structural prior to constrain latent space geometry, the method effectively disentangles noise. This approach integrates IB theory, structured latent variable modeling, and deep learning regularization techniques. It yields substantial improvements on low-data classification tasks while maintaining consistent performance gains across dense prediction benchmarks.

Abstract

Standard information bottleneck (IB) regularization constrains representations via a single scalar I(Z;X), implicitlytreating all information as homogeneous. However, a single global compression control couples label-relevant structurewith residual within-condition variation, rather than regulating their allocation independently, allowing nuisanceinformation to persist in learned representations. For example, in medical imaging applications, residual variation oftenstems from acquisition conditions, background factors, or subject-specific appearance. This issue becomes particularlypronounced in data-limited settings, where models tend to overfit such variation, hindering generalization. While existingregularization methods can stabilize training, control capacity, or shape representation geometry, they do not explicitlyseparate nuisance-like variation from task-supporting structure. To address this limitation, we revisit IB from a structuredperspective based on a label-induced partition, where condition-level structure and within-condition information playdistinct roles. This leads to a dual-bottleneck formulation: a standard KL term controls global information capacity, while aconditional KL term targets within-condition information. We show that the conditional KL admits an exact decompositioninto a within-condition information term and a prior-mismatch term, explaining its alignment with the design objective.With a simplex-structured conditional prior, the method provides controllable latent geometry and integrates seamlesslyinto existing pipelines. Experiments on classification and segmentation show the clearest gains in low-data classificationand consistent improvements across dense prediction benchmarks.

Problem

• Information Bottleneck
• Nuisance Variation
• Structured Decomposition
• Generalization
• Label-Induced Partitions

Innovation

• Information Bottleneck
• Dual-Bottleneck
• Label-Induced Partition
• Conditional KL Decomposition
• Simplex-Structured Prior