Technology Category
Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP ModelsHumans and AI: Human-in-the-loop Machine LearningMachine Learning: Evaluation and Analysis
VIDEE: Visual and Interactive Decomposition, Execution, and Evaluation of Text Analytics with Intelligent Agents
🤖 AI Summary
Entry-level data analysts struggle to perform advanced text analysis using professional NLP tools due to high technical barriers. Method: This paper proposes a human-in-the-loop text analytics system built upon a “decompose–execute–evaluate” closed-loop framework: (1) it introduces the first MCTS-based generative reasoning guidance with explicit human feedback integration; (2) it automatically synthesizes executable text analytics pipelines; and (3) it jointly leverages LLM-based automated evaluation and interactive visual validation. Contribution/Results: The system significantly lowers the NLP adoption barrier while enhancing interpretability and controllability of analysis. Quantitative experiments and a user study involving 24 participants—from novices to experts—demonstrate that non-expert users achieve >89% task success rates and exhibit a 42% improvement in error identification, markedly advancing usability, effectiveness, and trustworthiness of text analytics.
Technology Category
Application Category
📝 Abstract
Text analytics has traditionally required specialized knowledge in Natural Language Processing (NLP) or text analysis, which presents a barrier for entry-level analysts. Recent advances in large language models (LLMs) have changed the landscape of NLP by enabling more accessible and automated text analysis (e.g., topic detection, summarization, information extraction, etc.). We introduce VIDEE, a system that supports entry-level data analysts to conduct advanced text analytics with intelligent agents. VIDEE instantiates a human-agent collaroration workflow consisting of three stages: (1) Decomposition, which incorporates a human-in-the-loop Monte-Carlo Tree Search algorithm to support generative reasoning with human feedback, (2) Execution, which generates an executable text analytics pipeline, and (3) Evaluation, which integrates LLM-based evaluation and visualizations to support user validation of execution results. We conduct two quantitative experiments to evaluate VIDEE's effectiveness and analyze common agent errors. A user study involving participants with varying levels of NLP and text analytics experience -- from none to expert -- demonstrates the system's usability and reveals distinct user behavior patterns. The findings identify design implications for human-agent collaboration, validate the practical utility of VIDEE for non-expert users, and inform future improvements to intelligent text analytics systems.
Problem
Research questions and friction points this paper is trying to address.
Enables entry-level analysts to perform advanced text analytics without NLP expertise
Integrates human feedback with AI for text analysis pipeline generation
Provides visual evaluation tools for validating text analytics results
Innovation
Methods, ideas, or system contributions that make the work stand out.
Human-in-the-loop Monte-Carlo Tree Search algorithm
Generates executable text analytics pipeline
Integrates LLM-based evaluation and visualizations
🔎 Similar Papers
No similar papers found.
