A Light Bilevel Refinement Aligns Self-Supervised Representations for Stronger Task-Specific Learning

2026-09-27

AI Summary

This study addresses the misalignment between self-supervised pretraining and downstream task objectives, which leads to suboptimal fine-tuning, as well as the prohibitive computational cost of conventional bilevel optimization that hinders its scalability to large models. To this end, we propose BiSSLight, a framework that pioneers the integration of M-FAC implicit gradient approximation and LoRA parameter-efficient fine-tuning into bilevel optimization, achieving lightweight pretraining-finetuning objective alignment for architectures such as Vision Transformers. Experimental results demonstrate that BiSSLight significantly enhances downstream performance across multi-task scenarios while reducing computational overhead by over an order of magnitude compared to existing methods. Notably, the performance gains increase with model scale, establishing BiSSLight as a scalable paradigm for efficient alignment in large-scale models.

Abstract

Self-supervised pretraining learns representations that are broadly transferable across downstream tasks, yet direct fine-tuning can be suboptimal due to misalignment between self-supervised and downstream task objectives, potentially degrading pretrained features beneficial to the downstream task. The BiSSL framework addressed this by introducing a transitional training stage formulated as a bilevel optimization problem, in which the downstream task objective guides the self-supervised learning process in refining pretrained representations to better facilitate subsequent fine-tuning. However, BiSSL relies on conventional bilevel optimization solving techniques whose costly implicit hypergradient approximations render the method increasingly impractical for contemporary model architectures. To make it efficient and scalable, we introduce BiSSLight, which combines M-FAC-based implicit gradient approximation with parameter-efficient fine-tuning via LoRA, enabling efficient application at larger scales that were previously impractical. Evaluation across multiple downstream tasks and contemporary model architectures shows that BiSSLight consistently improves downstream performance, with gains becoming more pronounced as model size increases despite stronger baselines. The method is highly computationally efficient, reducing computation time by more than a factor of ten compared to its predecessor on a ViT-H backbone.

Problem

• Self-supervised learning
• Bilevel optimization
• Representation alignment
• Scalability
• Computational efficiency

Innovation

• Bilevel Optimization
• Implicit Gradient Approximation
• Parameter-Efficient Fine-Tuning
• LoRA
• Self-Supervised Learning