Incidents During Microservice Decomposition: A Case Study

2025-05-14

AI Summary

Microservice decomposition frequently triggers production incidents, compromising system reliability. Drawing on Carbon Health’s real-world migration experience, this study conducts root-cause analysis and qualitative case studies of 107 decomposition-related production events, systematically identifying six recurrent failure patterns—the first such taxonomy in the literature. We propose a “monolith-first modularization, then serviceification” resilient evolutionary paradigm: architectural decoupling is performed internally within the monolith prior to inter-service separation, thereby reducing inter-service dependency complexity. Empirical evaluation demonstrates that this approach significantly lowers initial inter-service failure rates and deployment rollback frequency. Our contributions are threefold: (1) the first comprehensive taxonomy of microservice decomposition failure causes; (2) empirical validation that modularization as an intermediate step enhances system robustness during decomposition; and (3) a reusable, low-risk methodology for industrial architecture evolution.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Multiagent Systems: Agent-Based Simulation and Emergent Behavior
• Machine Learning: Scalability of ML Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Software errors and incidents are inevitable in web based applications. Scalability challenges, increasing demand, and ongoing code changes can contribute to such failures. As software architectures evolve rapidly, understanding how and why incidents occur is crucial for enhancing system reliability. In this study, we introduce Carbon Health's software stack, share our microservices journey, and analyze 107 incidents. Based on these incidents, we share insights and lessons learned on microservice decomposition. Finally, we suggest that starting with monolithic modularization as an initial step toward microservice decomposition may help reduce incidents and contribute to building more resilient software.

Problem

• Analyzing causes of incidents in microservice decomposition
• Studying impact of monolithic modularization on reliability
• Providing insights from 107 real-world microservice incidents

Innovation

• Analyzed 107 incidents in microservices
• Proposed monolithic modularization first
• Shared insights on decomposition lessons