A Comparison of Conversational Models and Humans in Answering Technical Questions: the Firefox Case

2025-10-24

AI Summary

Despite growing interest in retrieval-augmented generation (RAG) for developer support, its real-world efficacy in open-source software (OSS) contexts remains empirically unvalidated. Method: This study conducts the first systematic, human-evaluated assessment of RAG for technical Q&A in a large-scale OSS project—Mozilla Firefox—comparing responses from domain experts, standard GPT, and RAG-GPT. A test set derived from authentic developer chat logs is evaluated by domain experts via double-blind annotation across three dimensions: helpfulness, comprehensiveness, and conciseness. Contribution/Results: RAG-GPT achieves significantly higher comprehensiveness than human experts (62.50% vs. 54.17%) and matches them closely in helpfulness (75.00% vs. 79.17%), albeit with higher redundancy. This work establishes the first empirical benchmark for RAG in OSS collaboration, demonstrating its viability to augment core maintainers—improving scalability of technical support without compromising response quality.

Technology Category

• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Knowledge Representation and Reasoning: Qualitative Reasoning
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Web Mining and Content Analysis: Community question answering

Abstract

The use of Large Language Models (LLMs) to support tasks in software development has steadily increased over recent years. From assisting developers in coding activities to providing conversational agents that answer newcomers' questions. In collaboration with the Mozilla Foundation, this study evaluates the effectiveness of Retrieval-Augmented Generation (RAG) in assisting developers within the Mozilla Firefox project. We conducted an empirical analysis comparing responses from human developers, a standard GPT model, and a GPT model enhanced with RAG, using real queries from Mozilla's developer chat rooms. To ensure a rigorous evaluation, Mozilla experts assessed the responses based on helpfulness, comprehensiveness, and conciseness. The results show that RAG-assisted responses were more comprehensive than human developers (62.50% to 54.17%) and almost as helpful (75.00% to 79.17%), suggesting RAG's potential to enhance developer assistance. However, the RAG responses were not as concise and often verbose. The results show the potential to apply RAG-based tools to Open Source Software (OSS) to minimize the load to core maintainers without losing answer quality. Toning down retrieval mechanisms and making responses even shorter in the future would enhance developer assistance in massive projects like Mozilla Firefox.

Problem

• Evaluating RAG effectiveness in answering technical Firefox questions
• Comparing human vs AI response quality on developer queries
• Assessing RAG's potential to reduce maintainer workload in OSS

Innovation

• Used Retrieval-Augmented Generation for technical assistance
• Enhanced GPT model with retrieval mechanisms
• Applied RAG to Open Source Software projects