Assessing the Use of AutoML for Data-Driven Software Engineering

2023-07-20

International Symposium on Empirical Software Engineering and Measurement

AI Summary

Addressing the acute shortage of AI/ML expertise in software engineering (SE), this study investigates the effectiveness and adoption barriers of AutoML for SE decision-making. Method: We systematically benchmark 12 state-of-the-art AutoML tools (e.g., H2O, Auto-sklearn, TPOT) on SE datasets and complement quantitative evaluation with surveys and expert interviews. Contribution/Results: Our empirical analysis reveals that AutoML-generated models achieve significantly higher average accuracy than manually tuned models on SE classification tasks. However, 83% of the tools lack automated feature engineering and deployment capabilities, and provide insufficient workflow support for non-ML experts—exposing a critical “pseudo end-to-end” limitation. The study identifies structural gaps in full-lifecycle automation and cross-role collaboration within current AutoML systems, thereby providing evidence-based insights and concrete design directions for next-generation AutoML tailored to SE contexts.

Abstract

Background. Due to the widespread adoption of Artificial Intelligence (AI) and Machine Learning (ML) for building software applications, companies are struggling to recruit employees with a deep understanding of such technologies. In this scenario, AutoML is soaring as a promising solution to fill the AI/ML skills gap since it promises to automate the building of end-to-end AI/ML pipelines that would normally be engineered by specialized team members. Aims. Despite the growing interest and high expectations, there is a dearth of information about the extent to which AutoML is currently adopted by teams developing AI/ML-enabled systems and how it is perceived by practitioners and researchers. Method. To fill these gaps, in this paper, we present a mixed-method study comprising a benchmark of 12 end-to-end AutoML tools on two SE datasets and a user survey with follow-up interviews to further our understanding of AutoML adoption and perception. Results. We found that AutoML solutions can generate models that outperform those trained and optimized by researchers to perform classification tasks in the SE domain. Also, our findings show that the currently available AutoML solutions do not live up to their names as they do not equally support automation across the stages of the ML development workflow and for all the team members. Conclusions. We derive insights to inform the SE research community on how AutoML can facilitate their activities and tool builders on how to design the next generation of AutoML technologies.

Problem

• AutoML
• software engineering decisions
• AI/ML talent shortage

Innovation

• AutoML Effectiveness
• Software Engineering Decisions
• Automation Limitations