Human and Machine: How Software Engineers Perceive and Engage with AI-Assisted Code Reviews Compared to Their Peers

2025-01-03

AI Summary

This study investigates how LLM-assisted code review affects software engineers’ cognitive, affective, and behavioral practices. Drawing on in-depth interviews with 20 professional engineers and thematic coding analysis, it provides the first systematic deconstruction of human factors in AI-augmented code review. Results show that, compared to traditional peer review, LLM tools reduce affective regulation demands but substantially increase cognitive load; adoption of LLM-generated feedback is primarily constrained by a dual bottleneck of “trust” and “contextual awareness.” The study identifies “explanability” and “context-awareness” as two critical design dimensions for improving feedback adoption rates. Furthermore, it proposes a socio-technical framework for human–AI collaboration in code review, grounded in empirical evidence. This framework offers actionable design guidelines for AI-augmented development tools and advances foundational understanding of human–LLM interaction in software engineering practice.

Technology Category

• Humans and AI: User Experience and Usability
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The integration of artificial intelligence (AI) continues to increase and evolve, including in software engineering (SE). This integration involves processes traditionally entrusted to humans, such as coding. However, the impact on socio-technical processes like code review remains underexplored. In this interview-based study (20 interviewees), we investigate how software engineers perceive and engage with Large Language Model (LLM)-assisted code reviews compared to human peer-led reviews. In this inherently human-centric process, we aim to understand how software engineers navigate the introduction of AI into collaborative workflows. We found that engagement in code review is multi-dimensional, spanning cognitive, emotional, and behavioral dimensions. The introduction of LLM-assisted review impacts some of these attributes. For example, there is less need for emotional regulation and coping mechanisms when dealing with an LLM compared to peers. However, the cognitive load sometimes is higher in dealing with LLM-generated feedback due to its excessive details. Software engineers use a similar sense-making process to evaluate and adopt feedback suggestions from their peers and the LLM. However, the LLM feedback adoption is constrained by trust and lack of context in the review. Our findings contribute to a deeper understanding of how AI tools are impacting SE socio-technical processes and provide insights into the future of AI-human collaboration in SE practices.

Problem

• AI-assisted code review
• software engineers
• comparison with traditional manual code review

Innovation

• AI-assisted Code Review
• Human-AI Interaction
• Software Engineering Workflow