Leveraging XP and CRISP-DM for Agile Data Science Projects

2025-05-27

AI Summary

Data science projects face challenges in reconciling agile development with structured methodologies, hindering cross-functional collaboration and model delivery speed. Method: This study conducts the first empirical investigation into integrating eXtreme Programming (XP) with CRISP-DM, based on a mixed-methods study of a real-world e-commerce team (qualitative interviews and surveys; N=XX). Contribution/Results: Findings reveal high co-adoption rates—86% frequently use CRISP-DM, and 71% implement XP practices—demonstrating practical compatibility. We propose an “Iterative Structured Framework” that retains CRISP-DM’s six-phase macro-lifecycle while embedding XP’s core practices (e.g., pair programming, continuous integration, user-story-driven development) to enable agile micro-deliveries. The framework yields a reusable methodology and enterprise-level implementation guidelines, significantly improving cross-functional coordination efficiency and model delivery responsiveness. This work provides both theoretical grounding and a practical paradigm for agile data science.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Web Mining and Content Analysis: Machine learning and data science for the Web
• User Modeling, Personalization and Recommendation: Practical large-scale studies of user experience
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

This study explores the integration of eXtreme Programming (XP) and the Cross-Industry Standard Process for Data Mining (CRISP-DM) in agile Data Science projects. We conducted a case study at the e-commerce company Elo7 to answer the research question: How can the agility of the XP method be integrated with CRISP-DM in Data Science projects? Data was collected through interviews and questionnaires with a Data Science team consisting of data scientists, ML engineers, and data product managers. The results show that 86% of the team frequently or always applies CRISP-DM, while 71% adopt XP practices in their projects. Furthermore, the study demonstrates that it is possible to combine CRISP-DM with XP in Data Science projects, providing a structured and collaborative approach. Finally, the study generated improvement recommendations for the company.

Problem

• Integrate XP agility with CRISP-DM in Data Science projects
• Assess team adoption of CRISP-DM and XP practices
• Provide structured collaborative approach for agile Data Science

Innovation

• Integrates XP and CRISP-DM for agility
• Uses case study with e-commerce data
• Combines structured and collaborative approaches