LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selection

2025-05-01

AI Summary

To address the challenge of efficient model selection amid the proliferation of open-source large language models (LLMs) and the diversity of downstream tasks, this paper proposes a dynamic fine-tuning behavior modeling framework. First, it introduces the Hessian matrix into the PAC-Bayes framework to derive a computationally tractable generalization bound, thereby uncovering the intrinsic dynamics of fine-tuning. Second, leveraging the Neural Tangent Kernel (NTK), it develops LENSLLM—a lightweight scaling model capable of accurately and efficiently predicting cross-task performance. Evaluated on three large-scale benchmarks, LENSLLM achieves a model selection accuracy of 91.1% while reducing computational cost by 88.5%, significantly outperforming five state-of-the-art methods. The code and experimental results are publicly available.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The proliferation of open-sourced Large Language Models (LLMs) and diverse downstream tasks necessitates efficient model selection, given the impracticality of fine-tuning all candidates due to computational constraints. Despite the recent advances in LLM selection, a fundamental research question largely remains nascent: how can we model the dynamic behaviors of LLMs during fine-tuning, thereby enhancing our understanding of their generalization performance across diverse downstream tasks? In this work, we propose a novel theoretical framework that provides a proper lens to assess the generalization capabilities of LLMs, thereby enabling accurate and efficient LLM selection for downstream applications. In particular, we first derive a Hessian-based PAC-Bayes generalization bound that unveils fine-tuning dynamics of LLMs and then introduce LENSLLM, a Neural Tangent Kernel(NTK)-based Rectified Scaling Model that enables accurate performance predictions across diverse tasks while maintaining computational efficiency. Extensive empirical results on 3 large-scale benchmarks demonstrate that our model achieves up to 91.1% accuracy and reduces up to 88.5% computational cost in LLM selection, outperforming 5 state-of-the-art methods. We open-source our proposed LENSLLM model and corresponding results at the Github link: https://github.com/Susan571/LENSLLM.git.

Problem

• Model dynamic behaviors of LLMs during fine-tuning
• Enhance understanding of generalization performance across tasks
• Enable accurate and efficient LLM selection for applications

Innovation

• Hessian-based PAC-Bayes generalization bound
• NTK-based Rectified Scaling Model
• Computationally efficient LLM selection