Test-driven Software Experimentation with LASSO: an LLM Benchmarking Example

2024-10-11

arXiv.org

AI Summary

Empirical software engineering lacks standardized tooling to support Test-Driven Software Experiments (TDSE)—i.e., executing subject software and observing/analyzing its runtime behavior. To address this gap, we propose LASSO, the first general-purpose TDSE framework explicitly designed for runtime semantic assessment. LASSO integrates a domain-specific language (DSL), executable experiment scripts, dynamic behavioral instrumentation and analysis infrastructure, and an open-source web platform with interactive visualization. It enables self-contained, reusable, and extensible empirical studies on the reliability of LLM-generated code. An empirical evaluation conducted using LASSO validates its effectiveness in facilitating rigorous, reproducible experimentation. The platform has been publicly released as open-source software and adopted in practice, demonstrably improving experimental development efficiency, reproducibility, and interpretability.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Empirical software engineering faces a critical gap: the lack of standardized tools for rapid development and execution of Test-Driven Software Experiments (TDSEs) - that is, experiments that involve the execution of software subjects and the observation and analysis of their"de facto"run-time behavior. In this paper we present a general-purpose analysis platform called LASSO that provides a minimal set of domain-specific languages and data structures to conduct TDSEs. By empowering users with an executable scripting language to design and execute TDSEs, LASSO enables efficient evaluation of run-time semantics and execution characteristics in addition to statically determined properties. We present an example TDSE that demonstrates the practical benefits of LASSO's scripting capabilities for assessing the reliability of LLMs for code generation by means of a self-contained, reusable and extensible study script. The LASSO platform is freely available at: https://softwareobservatorium.github.io/, and a demo video is available on YouTube: https://youtu.be/tzY9oNTWXzw

Problem

• Lack of standardized tools for Test-Driven Software Experiments
• Need for efficient evaluation of run-time semantics in software
• Assessing reliability of LLMs for code generation

Innovation

• General-purpose platform for Test-Driven Software Experiments
• Executable scripting language for designing TDSEs
• Enables efficient evaluation of run-time semantics