Vision-Centric Representation-Efficient Fine-Tuning for Robust Universal Foreground Segmentation

2025-04-20

AI Summary

Visual foundation models (VFMs) suffer from poor generalization under parameter-efficient fine-tuning (PEFT) in texture-sparse scenarios—e.g., camouflaged and infrared imagery—due to inherent texture bias. Method: We propose Ladder Shape-bias Representation Side-tuning (LSR-ST), the first PEFT method explicitly optimizing for representation efficiency via an information bottleneck–guided shape-preference induction mechanism. LSR-ST integrates HDConv blocks (large-kernel attention + residual learning), multi-order feature interaction, sparse connectivity, and explicit shape-bias priors to enhance lightweight shape representation. Contribution/Results: With only 4.719M trainable parameters, LSR-ST achieves consistent improvements across 17 datasets and six foreground segmentation tasks, significantly boosting robustness and generalization in texture-sparse environments.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Foreground segmentation is crucial for scene understanding, yet parameter-efficient fine-tuning (PEFT) of vision foundation models (VFMs) often fails in complex scenarios, such as camouflage and infrared imagery. We attribute this challenge to the inherent texture bias in VFMs, which is exacerbated during fine-tuning and limits generalization in texture-sparse environments. To address this, we propose Ladder Shape-bias Representation Side-tuning (LSR-ST), a lightweight PEFT framework that enhances model robustness by introducing shape-biased inductive priors. LSR-ST captures shape-aware features using a simple HDConv Block, which integrates large-kernel attention and residual learning. The method satisfies three key conditions for inducing shape bias: large receptive fields, multi-order feature interactions, and sparse connectivity. Our analysis reveals that these improvements stem from representation efficiency-the ability to extract task-relevant, structurally grounded features while minimizing redundancy. We formalize this concept via Information Bottleneck theory and advocate for it as a key PEFT objective. Unlike traditional NLP paradigms that focus on optimizing parameters and memory, visual tasks require models that extract task-defined semantics, rather than just relying on pre-encoded features. This shift enables our approach to move beyond conventional trade-offs, offering more robust and generalizable solutions for vision tasks. With minimal changes to SAM2-UNet, LSR-ST achieves consistent improvements across 17 datasets and 6 tasks using only 4.719M trainable parameters. These results highlight the potential of representation efficiency for robust and adaptable VFMs within complex visual environments.

Problem

• Enhance robustness in universal foreground segmentation
• Address texture bias in vision foundation models
• Improve shape-aware feature extraction efficiency

Innovation

• LSR-ST enhances robustness with shape-biased priors
• HDConv Block integrates large-kernel attention
• Representation efficiency minimizes redundancy in features