Self-Supervised Weight Templates for Scalable Vision Model Initialization

2026-01-27

AI Summary

This work proposes SWEET, a scalable framework for initializing vision models that overcomes the inflexibility of conventional pre-trained models in adapting to architectures of varying depth and width. SWEET leverages self-supervised learning to construct a shared-weight template together with lightweight, size-specific scalers, enabling modular parameter generation under Tucker tensor decomposition constraints. A key innovation is the introduction of a stochastic width-scaling strategy during training, which substantially enhances cross-width generalization. Extensive experiments demonstrate that SWEET achieves state-of-the-art performance across diverse tasks—including image classification, object detection, semantic segmentation, and generative modeling—while enabling efficient initialization and transfer of vision models at multiple scales.

Technology Category

• Computer Vision: Large Vision Models
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The increasing scale and complexity of modern model parameters underscore the importance of pre-trained models. However, deployment often demands architectures of varying sizes, exposing limitations of conventional pre-training and fine-tuning. To address this, we propose SWEET, a self-supervised framework that performs constraint-based pre-training to enable scalable initialization in vision tasks. Instead of pre-training a fixed-size model, we learn a shared weight template and size-specific weight scalers under Tucker-based factorization, which promotes modularity and supports flexible adaptation to architectures with varying depths and widths. Target models are subsequently initialized by composing and reweighting the template through lightweight weight scalers, whose parameters can be efficiently learned from minimal training data. To further enhance flexibility in width expansion, we introduce width-wise stochastic scaling, which regularizes the template along width-related dimensions and encourages robust, width-invariant representations for improved cross-width generalization. Extensive experiments on \textsc{classification}, \textsc{detection}, \textsc{segmentation} and \textsc{generation} tasks demonstrate the state-of-the-art performance of SWEET for initializing variable-sized vision models.

Problem

• scalable initialization
• vision models
• model scaling
• weight templates
• self-supervised pre-training

Innovation

• self-supervised learning
• scalable model initialization
• weight template
• Tucker decomposition
• width-wise stochastic scaling