Architecture, Dataset and Model-Scale Agnostic Data-free Meta-Learning

2023-03-20

Computer Vision and Pattern Recognition

AI Summary

Existing data-free meta-learning methods are constrained to parameter-space optimization and require homogeneous model architectures, limiting scalability to large-scale pretrained models. This paper introduces the first data-free meta-learning framework tailored for heterogeneous pretrained models, enabling extraction of implicit prior knowledge without access to original training data. Our approach features two core innovations: Episode Curriculum Inversion (ECI) and Inversion Calibration Following Inner Loop (ICFIL). Leveraging pseudo-task distillation, adversarial end-to-end meta-training, and curriculum-based pseudo-episode generation, the framework achieves generalizable meta-adaptation across architectural heterogeneity, model scales (up to 10B parameters), and diverse datasets. Experiments demonstrate substantial improvements over state-of-the-art data-free meta-learning methods across multiple benchmarks, validating both strong generalization and seamless scalability.

Abstract

The goal of data-free meta-learning is to learn useful prior knowledge from a collection of pre-trained models without accessing their training data. However, existing works only solve the problem in parameter space, which (i) ignore the fruitful data knowledge contained in the pretrained models; (ii) can not scale to large-scale pre-trained models; (iii) can only meta-learn pre-trained models with the same network architecture. To address those issues, we propose a unified framework, dubbed PURER, which contains: (1) ePisode cUrriculum inveRsion (ECI) during data-free meta training; and (2) invErsion calibRation following inner loop (ICFIL) during meta testing. During meta training, we propose ECI to perform pseudo episode training for learning to adapt fast to new unseen tasks. Specifically, we progressively synthesize a sequence of pseudo episodes by distilling the training data from each pre-trained model. The ECI adaptively increases the difficulty level of pseudo episodes according to the real-time feedback of the meta model. We formulate the optimization process of meta training with ECI as an adversarial form in an end-to-end manner. During meta testing, we further propose a simple plug-and-play supplement—ICFIL—only used during meta testing to narrow the gap between meta training and meta testing task distribution. Extensive experiments in various real-world scenarios show the superior performance of ours.

Problem

• Learn prior knowledge without training data
• Scale to large pre-trained models
• Meta-learn across different architectures

Innovation

• Data-free meta-learning framework
• Pseudo episode curriculum inversion
• Inversion calibration for task distribution