Do you monitor CI Practices? I don't know. You tell me: A case study

2025-03-04

AI Summary

This study investigates the practical value and challenges of Continuous Integration (CI) practice monitoring in software development across three Brazilian public-sector organizations. Addressing problems of low CI visibility and evidence-deficient process improvement, the authors designed and deployed a lightweight CI monitoring tool, conducting a multi-case mixed-methods empirical study comprising surveys, in-depth interviews, build log analysis, and CI platform data mining. To our knowledge, this is the first systematic, organization-level evaluation of CI monitoring in real-world government settings. Results demonstrate that monitoring significantly enhances CI visibility and quality perception, fosters team collaboration and continuous improvement, and exhibits low intrusiveness and high usability. The study further proposes embedding CI monitoring capabilities natively into CI service dashboards. It contributes a reusable methodology and empirically grounded insights to advance engineering effectiveness in the public sector.

Technology Category

• Computer Vision: Bias, Fairness & Privacy
• Constraint Satisfaction and Optimization: Solvers and Tools
• Cognitive Modeling & Cognitive Systems: Applications

Application Category

• Security and Privacy: Large-scale security measurements
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Web Mining and Content Analysis: Web measurements

Abstract

Background: In this paper we seek for understand the benefits and challenges of monitoring CI practices in day-to-day software development. Aims: We aim to evaluate the impact of monitoring seven CI practices in a real-world scenario on three public organizations in Brazil. Method: We first developed a CI practices monitoring suite tool and conducted a multiple-case study applying a mixed-methods strategy, combining surveys, interviews, log data, and mining data from CI services. Results: We verified organization' interest in monitoring CI practices. Monitoring provided an overview of the organization's CI status, not covered by other tools, motivated constant improvement in these practices, a perception of software quality, improve the communication and and it is easy to adopt. Conclusions: We recommend that companies adopt monitoring of CI practices and that CI services integrate this monitoring into their dashboards.

Problem

• Understanding benefits and challenges of CI monitoring
• Evaluating impact of monitoring CI practices in organizations
• Developing tools for CI practice monitoring and analysis

Innovation

• Developed CI practices monitoring suite tool
• Applied mixed-methods strategy for evaluation
• Integrated monitoring into CI service dashboards