🤖 AI Summary
This study addresses the persistent occurrence of user-visible defects in production despite passing automated tests, particularly in complex scenarios involving large language model ensembles, multi-market internationalization, and browser-based front-end compositions. To tackle this issue, the authors propose a “four-seam” defect escape analysis framework. Through root cause analysis of 252 defect-fixing commits, they identify that 44% of defects originate from four critical verification blind spots: real-browser runtime environments, non-default markets, end-to-end user workflows, and full-system integration levels. Building on these insights, the team developed an automated test suite encompassing 1,553 test cases and introduced an actionable method for prioritizing boundary testing. This approach significantly reduces the recurrence of similar defects in production deployments and effectively compensates for the coverage limitations of conventional unit testing.
📝 Abstract
Modern web applications increasingly combine three ingredients that are hard to test: output from large language models, multi-market internationalization, and browser-driven front-ends over external data sources. We report on a production rental-search assistant whose automated suite grew to 1,553 test cases in six weeks. The suite passed continuously, yet user-facing defects continued to reach production. We studied all 252 bug-fix commits in the project and classified each by the boundary, or seam, it escaped through. About 44 percent of the fixes fall in four seams that component-level unit tests cannot observe: the live browser runtime, the non-default market, the end-to-end flow, and the whole-system level. A fix without a guard at the seam let one defect ship twice. We present the four-seam framework, the measured defect distribution, and the practices we adopted, including a simple way for a team to find the seam that carries the most fixes.