Software Engineering for Large Language Models: Research Status, Challenges and the Road Ahead

2025-06-30

AI Summary

There is a lack of systematic, software engineering (SE)-driven analysis of the full lifecycle challenges associated with large language models (LLMs). Method: We conduct a structured systematic literature review grounded in the SE lifecycle model, decomposing LLM development into six phases—requirements, data, development, testing, deployment, and maintenance—and perform stage-wise problem identification. Contribution/Results: We uncover phase-specific SE challenges—including prompt engineering maintainability, insufficient test coverage under data drift, and misalignment between model versioning and code evolution—and propose the first SE-oriented research framework for LLMs. This framework delineates stage-specific research directions and integrated technical pathways. Our work bridges a critical theoretical gap at the intersection of LLMs and SE, delivering an actionable research roadmap and practical guidance for building efficient, reliable, and evolvable LLM-based software systems.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

The rapid advancement of large language models (LLMs) has redefined artificial intelligence (AI), pushing the boundaries of AI research and enabling unbounded possibilities for both academia and the industry. However, LLM development faces increasingly complex challenges throughout its lifecycle, yet no existing research systematically explores these challenges and solutions from the perspective of software engineering (SE) approaches. To fill the gap, we systematically analyze research status throughout the LLM development lifecycle, divided into six phases: requirements engineering, dataset construction, model development and enhancement, testing and evaluation, deployment and operations, and maintenance and evolution. We then conclude by identifying the key challenges for each phase and presenting potential research directions to address these challenges. In general, we provide valuable insights from an SE perspective to facilitate future advances in LLM development.

Problem

• Systematically analyze LLM development lifecycle challenges
• Identify key challenges in six SE-based LLM phases
• Propose research directions for LLM development solutions

Innovation

• Systematic analysis of LLM development lifecycle
• Identify key challenges in six phases
• Propose SE solutions for LLM advancement