Can LLMs Replace Humans During Code Chunking?

2025-06-24

AI Summary

Modernizing government legacy systems—particularly those written in low-resource, obsolete languages such as MUMPS and assembly language (ALC)—faces significant challenges due to ultra-long code contexts exceeding standard LLM context windows and high costs/inconsistency of manual code chunking. Method: We propose an autonomous, multi-model approach leveraging GPT-4o, Claude 3 Sonnet, Mixtral, and Llama 3 to perform semantic code chunking and generate module-level annotations for such legacy code. Contribution/Results: Experiments demonstrate that LLMs accurately identify semantic segmentation points aligned with domain experts’ judgments. The generated module annotations achieve 20% higher factual accuracy and 10% greater practical utility than human-authored counterparts. This work provides the first empirical validation of LLMs’ feasibility and superiority in understanding legacy code and generating structured documentation—establishing an efficient, scalable technical pathway for large-scale legacy system modernization.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Cognitive Modeling & Cognitive Systems: Analogy

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Large language models (LLMs) have become essential tools in computer science, especially for tasks involving code understanding and generation. However, existing work does not address many of the unique challenges presented by code written for government applications. In particular, government enterprise software is often written in legacy languages like MUMPS or assembly language code (ALC) and the overall token lengths of these systems exceed the context window size for current commercially available LLMs. Additionally, LLMs are primarily trained on modern software languages and have undergone limited testing with legacy languages, making their ability to understand legacy languages unknown and, hence, an area for empirical study. This paper examines the application of LLMs in the modernization of legacy government code written in ALC and MUMPS, addressing the challenges of input limitations. We investigate various code-chunking methods to optimize the generation of summary module comments for legacy code files, evaluating the impact of code-chunking methods on the quality of documentation produced by different LLMs, including GPT-4o, Claude 3 Sonnet, Mixtral, and Llama 3. Our results indicate that LLMs can select partition points closely aligned with human expert partitioning. We also find that chunking approaches have significant impact on downstream tasks such as documentation generation. LLM-created partitions produce comments that are up to 20% more factual and up to 10% more useful than when humans create partitions. Therefore, we conclude that LLMs can be used as suitable replacements for human partitioning of large codebases during LLM-aided modernization.

Problem

• Can LLMs handle legacy government code chunking?
• Do LLMs understand legacy languages like MUMPS and ALC?
• How does code-chunking affect documentation quality?

Innovation

• Optimizing code-chunking for legacy languages
• Evaluating LLMs on MUMPS and ALC
• LLMs improve documentation quality significantly