Asking What Matters: Reward-Driven Clarification for Software Engineering Tasks

2026-04-16

AI Summary

This work addresses the challenge of incomplete user requirements in software engineering tasks, which often hinders effective clarification by AI assistants. The authors propose CLARITI, an 8-billion-parameter clarification module that introduces Shapley attribution and distribution comparison into clarification strategy optimization for the first time. CLARITI employs a multi-stage reinforcement learning reward mechanism centered on information value and user answerability. Experimental results demonstrate that this approach significantly improves clarification efficiency, achieving a problem-resolution rate comparable to GPT-5 on underspecified software engineering tasks while reducing the number of clarification questions by 41%.

Technology Category

Natural Language Processing: Prompt Engineering / PromptingSearch and Optimization: Metareasoning and MetaheuristicsConstraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

Search and Retrieval-Augmented AI: Agentic searchUser Modeling, Personalization and Recommendation: Fairness-aware retrieval and rankingSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Humans often specify tasks incompletely, so assistants must know when and how to ask clarifying questions. However, effective clarification remains challenging in software engineering tasks as not all missing information is equally valuable, and questions must target information users can realistically provide. We study clarification in real software engineering tasks by quantifying which types of information most affect task success and which questions elicit useful responses from simulated users. Using Shapley attribution and distributional comparisons, we identify two key properties of effective clarification: task relevance (which information predicts success) and user answerability (what users can realistically provide). We operationalize these properties as multi-stage reinforcement learning rewards to train CLARITI, an 8B-parameter clarification module, that matches GPT-5's resolution rate on underspecified issues while generating 41% fewer questions. Our results suggest that grounding reward design in empirical analysis of information impact and user answerability improves clarification efficiency.

Problem

• clarification
• software engineering
• task specification
• user answerability
• task relevance

Innovation

• reward-driven clarification
• task relevance
• user answerability
• reinforcement learning
• Shapley attribution