What Are Developers Actually Discussing When Visual Regression Tests Fail?

2026-08-07

AI Summary

This study challenges the conventional view that visual regression testing (VRT) is solely for detecting styling issues by empirically analyzing 307 VRT-enabled pull requests using Chromatic alongside 299 non-VRT control requests. Through card sorting of 189 VRT-flagged issues, the work systematically demonstrates that VRT not only identifies visual defects—such as layout and color inconsistencies—but also uncovers 18.5% non-styling issues, including undefined states and missing content, as well as cross-file, non-local effects. The findings reveal that VRT-related discussions are ten times more frequent and resolution times 3.8 times longer than in non-VRT cases. Furthermore, the study delineates seven defect categories, significantly broadening the understanding of VRT’s role in software maintenance beyond mere visual validation.

Abstract

Visual Regression Tests (VRTs) are widely adopted as a mechanism for detecting unintended visual changes in user interfaces. By design, VRTs operate on rendered pixel output, and the prevailing assumption is that they catch stylistic regressions such as layout shifts, color mismatches, and font alterations. We conduct an empirical analysis of 307 pull requests (PRs) from 103 GitHub repositories that incorporate VRT results via Chromatic, comparing them against 299 PRs that contain image attachments but no VRT (Visual PRs). Quantitatively, VRT-PRs show no significant acceptance-rate difference, but exhibit a 3.8 times longer median resolution time, 10 times more discussion comments, and 1.75 to 4.5 times larger code changes than Visual PRs. VRT results are typically shared around the midpoint of the review process, sustaining ongoing discussion rather than serving only as a final check. Through a card-sorting analysis of 189 VRT-flagged issues, we identify seven defect categories assigned to the analyzed issues: Layout (39.7\%), Appearance (27.5\%), Color (14.8\%), Text (9.5\%), State (6.9\%), Test (6.3\%), and Image (4.2\%). The three most frequent categories are stylistic, while approximately 18.5\% of analyzed issues (35/189) involve non-stylistic origins, including undefined component state (13 cases), content disappearance (17 cases across multiple categories), and visually imperceptible regressions (5 cases). We further document cases in which VRT detected visual regressions originating from code changes in seemingly unrelated files, exposing non-local effects that no targeted test would have been written to catch. These observations indicate that, in addition to its primary role as a stylistic checker, VRT functions as a secondary detector of unintended consequences of code changes, with implications for how VRT should be integrated into the maintenance toolchain.

Problem

• Visual Regression Testing
• Developer Discussion
• Stylistic Regression
• Non-stylistic Defects
• Code Change Impact

Innovation

• Visual Regression Testing
• Empirical Study
• Non-local Effects
• Defect Categorization
• Code Review