Is One-Shot In-Context Learning Helpful for Data Selection in Task-Specific Fine-Tuning of Multimodal LLMs?

2026-03-30

AI Summary

This work addresses the challenge of balancing performance and computational cost in task-specific fine-tuning as data scales grow, where prevailing data selection strategies struggle to jointly account for sample importance, diversity, and inter-sample interactions. The authors propose CLIPPER, a training-free data selection framework that introduces in-context learning with single examples into data selection for the first time. By probing the responses of multimodal large language models to example-query pairs, CLIPPER disentangles parametric from world knowledge and constructs a core subset by matching the perplexity distribution of the original data. Requiring neither auxiliary scoring models nor heuristic clustering, CLIPPER achieves a 47% improvement in data efficiency on VRSBench and reduces training time by 37% on ScienceQA when applied to Qwen2.5-VL-7B and Llama-3.2-11B-Vision-Instruct, respectively, while matching the performance of full-data fine-tuning.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Computer Vision: Large Vision ModelsNatural Language Processing: (Large) Language Models

Application Category

Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingEconomics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labelingSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Injecting world knowledge into pretrained multimodal large language models (MLLMs) is essential for domain-specific applications. Task-specific fine-tuning achieves this by tailoring MLLMs to high-quality in-domain data but encounters scalability challenges as datasets grow, necessitating a trade-off between performance and computational overhead. Existing data selection methods rely on additional scoring models or heuristic clustering, failing to concentrate on both data importance and diversity. Moreover, both methods overlook the interplay among training samples. To address these limitations, we propose CLIPPER, a training-free data selection pipeline that separates parameter and world knowledge, and leverages in-context learning to probe model responses to different demonstration-query combinations. CLIPPER identifies coresets that mirror the original dataset's perplexity distribution, preserving critical samples while maintaining diversity. Experiments on two MLLMs and three datasets show that CLIPPER matches full fine-tuning performance with significantly lower costs: Qwen2.5-VL-7B attains 47% data efficiency on VRSBench, and Llama-3.2-11B-Vision-Instruct reduces ScienceQA training time by 37%.

Problem

• data selection
• multimodal LLMs
• task-specific fine-tuning
• in-context learning
• coreset

Innovation

• in-context learning
• data selection
• multimodal LLMs
• training-free
• coreset