FACT: A Simple and Efficient Framework for Active Finetuning

2026-06-01

AI Summary

This work addresses the challenge of full-parameter fine-tuning in active learning scenarios with scarce labeled data, where naive fine-tuning often distorts pretrained features and leads to overfitting. To mitigate this, the authors propose FACT, a novel framework that formally defines the task of Fine-grained Active Fine-tuning (FiAF) and introduces a three-stage hierarchical fine-tuning pipeline integrated with a Frozen Feature Augmentation (FroFA) mechanism to jointly optimize data selection and parameter update strategies. FACT is compatible with diverse backbones—including ConvNeXt, ViT, and ViL—and achieves state-of-the-art performance across benchmarks such as CIFAR-10/100 and ImageNet-1k, significantly enhancing few-shot generalization. Notably, under low sampling rates, it improves ViT accuracy by over 20%, demonstrating remarkable efficiency, robustness, and generalizability.

Abstract

The main goal of active finetuning is to improve a pretrained model's performance on a specific task or domain by finetuning it with carefully selected informative or challenging data. Previous research has predominantly focused on the active aspect (i.e., data selection) while uniformly employing full finetuning for model adaptation, which inevitably distorts pretrained features due to distribution shift. This issue becomes particularly pronounced when the model size is large relative to the finetuning data quantity, leading to heightened overfitting risks. To address this critical gap, we formally outline the FiAF task that emphasizes systematic exploration of finetuning methodologies in active learning. We propose FACT, a three-phase hierarchical finetuning framework featuring both efficiency and simplicity, specifically designed for active finetuning scenarios. Our comprehensive experiments span: (1) Three major dataset categories encompassing classic (CIFAR10, CIFAR100, ImageNet-1k), imbalanced (CIFAR10-LT, CIFAR100-LT), and fine-grained (StanfordCars, FGVCAircraft) image classification datasets, each evaluated under 3-5 distinct sampling ratios; (2) Diverse pretrained architectures including Convolutional Neural Network (ConvNeXt), Vision Transformer (ViT), and Vision LSTM (ViL) networks; (3) A systematic investigation of frozen feature augmentation (FroFA) strategies. (4) A comprehensive and rigorous analysis of efficiency and generalizability. The results demonstrate significant improvements with strong generalization and robustness. Notably, under low sampling ratios, our framework achieves remarkable performance gains of over 20% on the ViT model for CIFAR10, CIFAR100, and ImageNet-1k benchmarks. This systematic approach establishes new state-of-the-art performance while maintaining parameter efficiency, proving particularly effective when labeled data is scarce.

Problem

• active finetuning
• distribution shift
• overfitting
• pretrained models
• data scarcity

Innovation

• active finetuning
• parameter-efficient finetuning
• frozen feature augmentation
• distribution shift mitigation
• hierarchical finetuning framework