BMFA: Boundary-Minority Free-Energy Adaptive Screening

2026-07-20

AI Summary

This work addresses a critical limitation of Vision Transformers, which tend to overlook spatially small yet strongly activated boundary regions during patch-wise mean aggregation, leading to ineffective attention mechanisms. The study formally characterizes this underestimation phenomenon for the first time and introduces a computable selection criterion grounded in a lower bound of local free energy increments. By leveraging hierarchical piecewise-constant approximations with recursive refinement, the method adaptively preserves salient regions. It overcomes the constraints of finite-order moment correction, substantially reducing underestimation error from 2.582 to 0.261 on synthetic data (with a leaf-node rate of 5.794%) and narrowing the COCO boundary gap from 2.254 to 0.526, while maintaining a Top-1 accuracy of 71.520% on ImageNet-1K at a leaf-node rate of 55.861%.

Abstract

Vision Transformers process spatially redundant tokens efficiently only when coarse token summaries preserve the evidence required by exponential attention aggregation. We identify a boundary-minority underestimation failure in which a spatially small, high-response region contributes dominant Gibbs mass while remaining nearly invisible to a block mean. We formalize the failure through the discrepancy between normalized log-mean-exp free energy and mean summarization, prove that minority Gibbs mass can remain non-vanishing as its spatial support and mean contribution vanish, and characterize the limitations of finite-order moment corrections. Building on the resulting analysis, we introduce Boundary-Minority Free-Energy Adaptive Screening (BMFA), which constructs a hierarchical piecewise-constant approximation and recursively refines blocks according to a computable lower-bound increment of local free energy. Controlled synthetic tests, COCO and LVIS diagnostic probes, closed-loop DeiT-Tiny evaluations, and ImageNet-1K experiments establish a consistent evidence chain. BMFA reduces the mean synthetic underestimate from 2.582 to 0.261 at a 5.794% leaf ratio, lowers the COCO image-edge mean gap from 2.254 to 0.526, and preserves 71.520% ImageNet Top-1 accuracy at a 55.861% leaf ratio. The current prototype evaluates selection quality after full QK computation; the reported leaf ratio therefore characterizes representation granularity rather than verified sparse-kernel speedup.

Problem

• Boundary-Minority
• Free-Energy
• Token Summarization
• Attention Aggregation
• Vision Transformers

Innovation

• Free Energy
• Vision Transformers
• Adaptive Token Screening
• Boundary-Minority Estimation
• Hierarchical Approximation