EmpaAva: An Open-source Agentic 3D-Avatar Empathetic Live Chatbot

2026-08-05

AI Summary

This work extends empathetic dialogue from purely textual interactions to real-time, face-to-face conversations with 3D virtual agents by introducing the first open-source, end-to-end controllable multimodal empathy system. Built upon a Tri-Agent architecture orchestrated by a large language model, the system integrates multimodal emotion perception, affective response planning, lip-syncing, and 3D Gaussian rendering to achieve coherent empathetic expression across speech, facial expressions, and gestures. Experimental results demonstrate that the proposed system significantly outperforms text-only baselines, 2D avatars, and existing virtual human approaches in emotional understanding, response quality, and audiovisual consistency.

Abstract

This paper presents EmpaAva, to our knowledge the first open-source, agentic 3D-avatar empathetic chatbot, which carries empathetic response generation (ERG) from text-only exchanges into live, face-to-face interaction. Through a video-call-like interface, a user speaks to a 3D digital human that reads their affect from speech and optional vision, and replies with emotional speech, lip-synced facial motion, and photorealistic 3D Gaussian rendering. At its core, an LLM coordinates a Tri-Agent Architecture, in which perception, empathetic response planning, and embodied rendering form a closed loop, paired with a Response Planning layer that compiles each reply into an executable multimodal plan, keeping voice, expression, and rendering on one empathetic intent. Building on strong open-source modules, EmpaAva supplies the intelligence that binds them into one controllable, inspectable experience. In automatic and human evaluations, EmpaAva surpasses text-only, 2D talking-face, and multimodal avatar baselines in emotion understanding, response quality, and audio-visual consistency. We open-source EmpaAva with an online live demo.

Problem

• empathetic response generation
• 3D avatar
• multimodal interaction
• affective computing
• live chatbot

Innovation

• agentic 3D-avatar
• empathetic response generation
• Tri-Agent Architecture
• multimodal planning
• 3D Gaussian rendering