Debiasing Architectural Decision-Making: An Experiment With Students and Practitioners

2025-02-06

AI Summary

This study investigates the impact of cognitive biases—specifically anchoring and optimism biases—on decision quality in software architecture. To address this, we designed and empirically evaluated a cognitive debiasing workshop tailored for practicing software architects. Employing a controlled experimental design, we integrated think-aloud protocols, qualitative coding, and quantitative measurement of bias-related behaviors—the first such validation of debiasing interventions in software architecture. Results demonstrate that the workshop significantly improves argumentation quality and increases the frequency of debiasing technique application, while substantially reducing the incidence of both anchoring and optimism biases. Notably, practitioners exhibited greater bias susceptibility and stronger intervention effects than students. Our key contribution is a reusable, evidence-based debiasing intervention framework grounded in cognitive psychology, offering empirically validated support for both architectural education and real-world practice.

Technology Category

• Cognitive Modeling & Cognitive Systems: (Computational) Cognitive Architectures
• Computer Vision: Bias, Fairness & Privacy
• Knowledge Representation and Reasoning: Preferences

Application Category

• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Cognitive biases are predictable, systematic errors in human reasoning. They influence decision-making in various areas, including architectural decision-making, where architects face many choices. For example, anchoring can cause architects to unconsciously prefer the first architectural solution that they came up with, without considering any solution alternatives. Prior research suggests that training individuals in debiasing techniques during a practical workshop can help reduce the impact of biases. The goal of this study was to design and evaluate a debiasing workshop with individuals at various stages of their professional careers. To test the workshop's effectiveness, we performed an experiment with 16 students and 20 practitioners, split into control and workshop group pairs. We recorded and analyzed their think-aloud discussions about improving the architectures of systems they collaborated on. The workshop improved the participants' argumentation when discussing architectural decisions and increased the use of debiasing techniques taught during the workshop. This led to the successful reduction of the researched biases' occurrences. In particular, anchoring and optimism bias occurrences decreased significantly. We also found that practitioners were more susceptible to cognitive biases than students, so the workshop had a more substantial impact on practitioners. We assume that the practitioners' attachment to their systems may be the cause of their susceptibility to biases. Finally, we identified factors that may reduce the effectiveness of the debiasing workshop. On that basis, we prepared a set of teaching suggestions for educators. Overall, we recommend using this workshop to educate both students and experienced practitioners about the typical harmful influences of cognitive bias on architectural decisions and how to avoid them.

Problem

• Reducing cognitive biases in architectural decision-making.
• Evaluating effectiveness of debiasing workshops on professionals.
• Comparing bias susceptibility between students and practitioners.

Innovation

• Designed debiasing workshop for architects
• Used think-aloud discussions for analysis
• Reduced anchoring and optimism biases