Brain-Inspired Stepwise Patch Merging for Vision Transformers

2024-09-11

arXiv.org

AI Summary

Vision Transformers (ViTs) suffer from weak hierarchical modeling capability in conventional Patch Merging, struggling to jointly capture global dependencies and local details. To address this, we propose a brain-inspired Stepwise Patch Merging (SPM) paradigm. SPM decouples global integration from local refinement: a learnable Multi-Scale Aggregation (MSA) module formalizes the brain’s multi-scale fusion mechanism to enhance long-range modeling; a Guided Local Enhancement (GLE) module dynamically strengthens salient local structures. Integrated into hierarchical ViT backbones, SPM supports end-to-end joint training. Extensive experiments on ImageNet-1K, COCO, and ADE20K demonstrate that SPM significantly improves performance on dense prediction tasks—achieving state-of-the-art accuracy and robustness in object detection and semantic segmentation.

Technology Category

• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Vertical and domain-specific search
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The hierarchical architecture has become a mainstream design paradigm for Vision Transformers (ViTs), with Patch Merging serving as the pivotal component that transforms a columnar architecture into a hierarchical one. Drawing inspiration from the brain's ability to integrate global and local information for comprehensive visual understanding, we propose a novel technique called Stepwise Patch Merging (SPM), which enhances the subsequent attention mechanism's ability to 'see' better. SPM comprises two critical modules: Multi-Scale Aggregation (MSA) and Guided Local Enhancement (GLE). The MSA module integrates multi-scale features to enrich feature representation, while the GLE module focuses on refining local detail extraction, thus achieving an optimal balance between long-range dependency modeling and local feature enhancement. Extensive experiments conducted on benchmark datasets, including ImageNet-1K, COCO, and ADE20K, demonstrate that SPM significantly improves the performance of various models, particularly in dense prediction tasks such as object detection and semantic segmentation. These results underscore the efficacy of SPM in enhancing model accuracy and robustness across a wide range of computer vision tasks.

Problem

• Enhances Vision Transformers' hierarchical architecture via brain-inspired patch merging
• Balances long-range dependency modeling and local feature enhancement
• Improves performance in dense prediction tasks like object detection

Innovation

• Brain-inspired Stepwise Patch Merging (SPM)
• Multi-Scale Aggregation and Guided Local Enhancement
• Enhances attention mechanism for better vision tasks