SliceFine: The Universal Winning-Slice Hypothesis for Pretrained Networks

2025-10-09

AI Summary

This work addresses the fundamental question in parameter-efficient fine-tuning (PEFT) of pretrained models: *why does fine-tuning a randomly sampled subnetwork (i.e., a “slice”) suffice?* We propose the **Universal Winning Slice Hypothesis** and establish the first theoretical framework proving that pretrained networks universally contain high-energy, spectrally balanced substructures well-suited to downstream tasks. Methodologically, we design a fixed-weight, random slicing strategy grounded in spectral analysis and representation redundancy—introducing no additional parameters. Our theory identifies two key phenomena: (i) task-relevant energy concentrates in low-rank spectral components, and (ii) effective slices exhibit spectral balance—a novel property ensuring stable optimization and generalization. Empirically, our approach matches or exceeds state-of-the-art PEFT methods across diverse language and vision benchmarks, while substantially improving training speed, memory efficiency, and model compactness.

Technology Category

• Machine Learning: Mixture of Experts (MoE)
• Computer Vision: Large Vision Models
• Search and Optimization: Learning to Search

Application Category

• Web Mining and Content Analysis: Large pretrained models with web data
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

This paper presents a theoretical framework explaining why fine tuning small, randomly selected subnetworks (slices) within pre trained models can be sufficient for downstream adaptation. We prove that pretrained networks exhibit a universal winning slice property arising from two phenomena: (1) spectral balance the eigenspectra of different weight matrix slices are remarkably similar; and (2) high task energy their backbone representations retain rich, task relevant features. This leads to the Universal Winning Slice Hypothesis, which provides a theoretical foundation for parameter efficient fine tuning (PEFT) in large scale models. Inspired by this, we propose SliceFine, a PEFT method that exploits this inherent redundancy by updating only selected slices of the original weights introducing zero new parameters, unlike adapter-based approaches. Empirically, SliceFine matches the performance of state of the art PEFT methods across language and vision tasks, while significantly improving training speed, memory efficiency, and model compactness. Our work bridges theory and practice, offering a theoretically grounded alternative to existing PEFT techniques.

Problem

• Explains why fine-tuning small subnetworks in pretrained models works effectively
• Proves pretrained networks have universal winning slice property from spectral balance
• Proposes parameter-efficient fine-tuning method improving speed and memory usage

Innovation

• Fine-tunes small random subnetworks in pretrained models
• Updates selected weight slices without adding parameters
• Improves training speed and memory efficiency significantly