TechCoach: Towards Technical Keypoint-Aware Descriptive Action Coaching

📅 2024-11-26
🏛️ arXiv.org
📈 Citations: 5
Influential: 0
📄 PDF
🤖 AI Summary
Existing action assessment methods yield only coarse-grained quality scores, lacking interpretable, fine-grained technical feedback. To address this, we introduce the Descriptive Action Coaching (DAC) task—first proposed for action skill instruction—that jointly generates a quality score and human-interpretable improvement suggestions spanning both keypoint-level and holistic-action levels. We construct EE4D-DAC, the first benchmark dataset for DAC; design a keypoint-aware hierarchical coaching paradigm comprising a context-aware keypoint reasoner, a keypoint-informed unified action evaluator, and a multimodal prompt fusion mechanism; and employ an LLM-driven multi-stage annotation pipeline to ensure high-quality labels. Experiments show our method significantly outperforms baselines on EE4D-DAC (BLEU-4 +12.3, ROUGE-L +9.7), yields feedback highly consistent with expert judgments, and enables interpretable quality attribution and precise motion correction.

Technology Category

Application Category

📝 Abstract
To guide a learner to master the action skills, it is crucial for a coach to 1) reason through the learner's action execution and technical keypoints, and 2) provide detailed, understandable feedback on what is done well and what can be improved. However, existing score-based action assessment methods are still far from this practical scenario. To bridge this gap, we investigate a new task termed Descriptive Action Coaching (DAC) which requires a model to provide detailed commentary on what is done well and what can be improved beyond a quality score from an action execution. To this end, we construct a new dataset named EE4D-DAC. With an LLM-based annotation pipeline, our dataset goes beyond the existing action assessment datasets by providing the hierarchical coaching commentary at both keypoint and instance levels. Furthermore, we propose TechCoach, a new framework that explicitly incorporates keypoint-level reasoning into the DAC process. The central to our method lies in the Context-aware Keypoint Reasoner, which enables TechCoach to learn keypoint-related quality representations by querying visual context under the supervision of keypoint-level coaching commentary. Prompted by the visual context and the keypoint-related quality representations, a unified Keypoint-aware Action Assessor is then employed to provide the overall coaching commentary together with the quality score. Combining all of these, we build a new benchmark for DAC and evaluate the effectiveness of our method through extensive experiments. Data and code will be publicly available.
Problem

Research questions and friction points this paper is trying to address.

Develop TechPoint-aware coaching for detailed action feedback
Create dataset EE4D-DescCoach with TechPoint-level commentary
Propose TechCoach framework for TechPoint-level reasoning in coaching
Innovation

Methods, ideas, or system contributions that make the work stand out.

Develops TechCoach framework for action coaching
Introduces Context-aware TechPoint Reasoner
Creates EE4D-DescCoach dataset with TechPoint commentary
🔎 Similar Papers
No similar papers found.