ChatHuman: Chatting about 3D Humans with Tools

2024-05-07

AI Summary

Existing 3D human analysis methods—spanning pose estimation, shape reconstruction, human-object interaction, and affective state inference—rely heavily on domain-specific expertise, limiting accessibility for non-specialists. This work introduces the first language-driven 3D human analysis system, enabling natural-language-based interactive invocation and interpretation of diverse specialized tools. Our approach integrates a large language model (LLM) with a tool orchestration framework and 3D geometric-semantic mapping. Key contributions include: (1) an LLM tool-teaching mechanism grounded in academic literature, enabling structured injection of tool knowledge; (2) retrieval-augmented generation (RAG)-driven zero-shot tool adaptation; and (3) a 3D output semantic translation module that unifies heterogeneous outputs into interpretable, semantically coherent representations. Evaluated across multiple tasks, the system achieves significantly higher tool selection accuracy and end-to-end analysis performance than prior methods, supporting real-time, explainable, and interactive 3D human analysis.

Technology Category

• Computer Vision: 3D Computer Vision
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Numerous methods have been proposed to detect, estimate, and analyze properties of people in images, including 3D pose, shape, contact, human-object interaction, and emotion. While widely applicable in vision and other areas, such methods require expert knowledge to select, use, and interpret the results. To address this, we introduce ChatHuman, a language-driven system that integrates the capabilities of specialized methods into a unified framework. ChatHuman functions as an assistant proficient in utilizing, analyzing, and interacting with tools specific to 3D human tasks, adeptly discussing and resolving related challenges. Built on a Large Language Model (LLM) framework, ChatHuman is trained to autonomously select, apply, and interpret a diverse set of tools in response to user inputs. Our approach overcomes significant hurdles in adapting LLMs to 3D human tasks, including the need for domain-specific knowledge and the ability to interpret complex 3D outputs. The innovations of ChatHuman include leveraging academic publications to instruct the LLM on tool usage, employing a retrieval-augmented generation model to create in-context learning examples for managing new tools, and effectively discriminating between and integrating tool results by transforming specialized 3D outputs into comprehensible formats. Experiments demonstrate that ChatHuman surpasses existing models in both tool selection accuracy and overall performance across various 3D human tasks, and it supports interactive chatting with users. ChatHuman represents a significant step toward consolidating diverse analytical methods into a unified, robust system for 3D human tasks.

Problem

• Integrates specialized 3D human analysis tools into a unified framework
• Enables non-experts to use complex 3D human analysis methods
• Overcomes LLM adaptation challenges for domain-specific 3D tasks

Innovation

• Integrates specialized methods into unified framework
• Uses LLM to autonomously select and apply tools
• Transforms 3D outputs into comprehensible formats