Can Large Language Models Improve SE Active Learning via Warm-Starts?

2024-12-30

AI Summary

This study addresses the challenge of low active learning efficiency in software engineering (SE) due to scarce labeled data in multi-objective SE tasks. We systematically investigate, for the first time, the efficacy of large language models (LLMs) as warm-start generators for active learning. Empirical evaluation across 49 SE tasks compares LLMs against Gaussian process models (GPMs) and tree-structured Parzen estimators (TPEs) in generating high-quality initial query points. Results show that LLMs significantly accelerate convergence and improve final performance on low- and medium-dimensional tasks—achieving an average speedup of 37% over state-of-the-art Bayesian methods. However, GPMs remain superior on high-dimensional tasks, revealing a clear dimensional boundary for LLM-based warm-start applicability. This work establishes the first large-scale empirical benchmark for LLM-augmented active learning in SE and provides interpretable, dimension-aware guidelines for practical deployment.

Technology Category

• Machine Learning: Active Learning
• Search and Optimization: Sampling/Simulation-based Search
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

When SE data is scarce,"active learners"use models learned from tiny samples of the data to find the next most informative example to label. In this way, effective models can be generated using very little data. For multi-objective software engineering (SE) tasks, active learning can benefit from an effective set of initial guesses (also known as"warm starts"). This paper explores the use of Large Language Models (LLMs) for creating warm-starts. Those results are compared against Gaussian Process Models and Tree of Parzen Estimators. For 49 SE tasks, LLM-generated warm starts significantly improved the performance of low- and medium-dimensional tasks. However, LLM effectiveness diminishes in high-dimensional problems, where Bayesian methods like Gaussian Process Models perform best.

Problem

• Large Language Models
• Software Engineering
• Active Learning

Innovation

• Large Language Models
• Active Learning
• Software Engineering