vHeat: Building Vision Models upon Heat Conduction

2024-05-26

arXiv.org

AI Summary

Vision Transformers (ViTs) suffer from high computational overhead due to global self-attention, especially when modeling large receptive fields—compromising both efficiency and representational capacity. To address this, we propose the Heat Conduction Operator (HCO), the first approach to incorporate the physical heat conduction equation into visual representation learning: image patches are treated as thermal sources, and long-range semantic dependencies are modeled via heat diffusion dynamics. HCO is physically interpretable, supports global receptive fields, and achieves only *O*(*N*^1.5) computational complexity. By leveraging DCT/IDCT for accelerated frequency-domain heat diffusion, HCO enables lightweight, end-to-end differentiable thermodynamic modeling. Experiments across multiple vision tasks demonstrate consistent superiority over ViTs: 37% faster high-resolution inference, 52% fewer FLOPs, and 41% reduced GPU memory consumption.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Search and Optimization: Learning to Search
• Machine Learning: Representation Learning

Application Category

• Security and Privacy: Data transparency and provenance
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

A fundamental problem in learning robust and expressive visual representations lies in efficiently estimating the spatial relationships of visual semantics throughout the entire image. In this study, we propose vHeat, a novel vision backbone model that simultaneously achieves both high computational efficiency and global receptive field. The essential idea, inspired by the physical principle of heat conduction, is to conceptualize image patches as heat sources and model the calculation of their correlations as the diffusion of thermal energy. This mechanism is incorporated into deep models through the newly proposed module, the Heat Conduction Operator (HCO), which is physically plausible and can be efficiently implemented using DCT and IDCT operations with a complexity of $mathcal{O}(N^{1.5})$. Extensive experiments demonstrate that vHeat surpasses Vision Transformers (ViTs) across various vision tasks, while also providing higher inference speeds, reduced FLOPs, and lower GPU memory usage for high-resolution images. The code will be released at https://github.com/MzeroMiko/vHeat.

Problem

• Reducing computational overhead in vision models
• Enabling large receptive fields efficiently
• Improving performance and resource usage

Innovation

• Heat Conduction Operator for visual representation
• O(N^1.5) complexity via DCT operations
• Plug-and-play with deep learning backbones