MathBuddy: A Multimodal System for Affective Math Tutoring

2025-08-27

AI Summary

Existing educational dialogue systems predominantly rely on large language models (LLMs) while neglecting the critical impact of student affect on learning outcomes. This work proposes the first multimodal affect-aware dialogue system for mathematics tutoring, integrating real-time facial expression analysis with conversational text to infer students’ emotional states. These states are mapped onto an eight-dimensional pedagogical strategy space to enable emotion-driven adaptive responses and closed-loop affective feedback. Our key contribution is the first deep integration of multimodal affect perception with LLMs, establishing a unified “perceive–model–respond” framework. Experimental results demonstrate that our system achieves a 23-percentage-point improvement in human preference win rate over baseline systems and attains a +3-point gain in the DAMR (Dialogue Affect Modeling and Response) composite score, significantly enhancing both instructional effectiveness and empathic capability.

Technology Category

• Cognitive Modeling & Cognitive Systems: Affective Computing
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Conversational AI/Dialog Systems

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

The rapid adoption of LLM-based conversational systems is already transforming the landscape of educational technology. However, the current state-of-the-art learning models do not take into account the student's affective states. Multiple studies in educational psychology support the claim that positive or negative emotional states can impact a student's learning capabilities. To bridge this gap, we present MathBuddy, an emotionally aware LLM-powered Math Tutor, which dynamically models the student's emotions and maps them to relevant pedagogical strategies, making the tutor-student conversation a more empathetic one. The student's emotions are captured from the conversational text as well as from their facial expressions. The student's emotions are aggregated from both modalities to confidently prompt our LLM Tutor for an emotionally-aware response. We have effectively evaluated our model using automatic evaluation metrics across eight pedagogical dimensions and user studies. We report a massive 23 point performance gain using the win rate and a 3 point gain at an overall level using DAMR scores which strongly supports our hypothesis of improving LLM-based tutor's pedagogical abilities by modeling students' emotions.

Problem

• Modeling student emotions in math tutoring
• Bridging affective states and pedagogical strategies
• Enhancing empathetic tutor-student conversations

Innovation

• Multimodal emotion capture from text and face
• Dynamic emotional mapping to teaching strategies
• Emotionally-aware LLM prompting for empathetic responses