Providing Information About Implemented Algorithms Improves Program Comprehension: A Controlled Experiment

2025-04-27

AI Summary

Existing automated algorithm detection techniques lack empirical validation of their practical utility in program comprehension. Method: We conducted a controlled experiment with 56 developers to assess the impact of algorithm-level semantic labels—comprising algorithm names and associated semantic information—on code understanding, comparing performance with and without such labels. Contribution/Results: Labels significantly improved comprehension accuracy (median increase of +6 points, ≈23%; *p* = 0.040, Mann–Whitney *U* test), particularly for developers with intermediate experience, without increasing comprehension time; 85% of participants reported that labels aided intent recognition. Through mixed-method analysis—quantitative (nonparametric hypothesis testing) and qualitative (thematic coding)—we systematically identified concrete benefits of algorithm labels in error detection, performance optimization, and library substitution scenarios. This work provides the first empirical foundation for algorithm-aware program understanding and semantic annotation research.

Technology Category

• Search and Optimization: Algorithm Configuration
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Other Foundations of Human Computation & AI

Application Category

• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Context: Various approaches aim to support program comprehension by automatically detecting algorithms in source code. However, no empirical evaluations of their helpfulness have been performed. Objective: To empirically evaluate how algorithm labels - which include the algorithm's name and additional information - impact program comprehension in terms of correctness and time. Method: We conducted a controlled experiment with 56 participants, where the experimental group received code with labeled algorithms. The groups completed exercises designed to measure program comprehension as well as a post-questionnaire on label helpfulness, use cases for algorithm recognition, and reasons for self-implementation of algorithms in practice. Results: Annotating source code with algorithm labels significantly improves program comprehension (p=0.040), with a median improvement of 6 points (~23%), but does not affect completion times (p=0.991). Qualitative analysis revealed that a majority of participants perceived the labels as helpful, especially for recognizing the codes intent. Participants also proposed use cases such as error detection, optimization, and library replacement. Reasons for self-implementing algorithms included library inadequacies, performance needs and avoiding dependencies or licensing costs. Conclusion: This study shows that algorithm labels improve program comprehension, especially for developers with medium programming experience. Our qualitative analysis also sheds light on how participants benefit from the labels, further use cases for algorithm recognition and motivations behind self-implementing algorithms.

Problem

• Evaluates impact of algorithm labels on program comprehension
• Measures correctness and time effects of labeled algorithms
• Identifies use cases and benefits of algorithm recognition

Innovation

• Algorithm labels improve program comprehension significantly
• Controlled experiment with 56 participants validates effectiveness
• Labels aid intent recognition and error detection