Coding Reliable LLM-based Integrated Task and Knowledge Agents with GenieWorksheets

2024-07-08

AI Summary

Current LLM-based dialogue agents face three key challenges in knowledge-intensive, task-oriented dialogues: frequent hallucinations, weak conditional logic reasoning, and difficulty integrating heterogeneous knowledge sources—leading to low reliability. This paper introduces Genie, a framework accompanied by the declarative GenieWorksheet specification, which enables robust parsing of complex conditional instructions and consistent multi-source knowledge integration via controllable strategy programming, strong knowledge grounding, and collaborative LLM execution. Our approach unifies programmable architecture design with declarative policy modeling to significantly improve instruction-following stability. On the STARV2 benchmark, Genie outperforms prior state-of-the-art by 20.5%. In real-user experiments, it achieves 21.1%, 20.1%, and 61% improvements over GPT-4+Function Calling in execution accuracy, dialogue act accuracy, and goal completion rate, respectively—demonstrating the effectiveness and practicality of highly reliable task–knowledge fusion agents.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Agentic search
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI

Abstract

Large Language Models (LLMs) present an opportunity to create automated assistants that can help users navigate complex tasks. However, existing approaches have limitations in handling conditional logic, integrating knowledge sources, and consistently following instructions. Researchers and industry professionals often employ ad hoc pipelines to construct conversational agents. These pipelines aim to maintain context, address failure cases, and minimize hallucinations, yet frequently fail to achieve these objectives. To this end, we present Genie - a programmable framework for creating task-oriented conversational agents that are designed to handle complex user interactions and knowledge queries. Unlike LLMs, Genie provides reliable grounded responses, with controllable agent policies through its expressive specification, Genie Worksheet. In contrast to dialog trees, it is resilient to diverse user queries, helpful with knowledge sources, and offers ease of programming policies through its declarative paradigm. The agents built using Genie outperforms the state-of-the-art method on complex logic domains in STARV2 dataset by up to 20.5%. Additionally, through a real-user study involving 62 participants, we show that Genie beats the GPT-4 with function calling baseline by 21.1%, 20.1%, and 61% on execution accuracy, dialogue act accuracy, and goal completion rate, respectively, on three diverse real-world domains

Problem

• Address LLM hallucination in conversational agents
• Improve conditional logic in task-oriented dialogues
• Integrate multi-source knowledge reliably

Innovation

• Programmable framework for conversational agents
• Advanced dialogue state management
• Declarative specification for controllable policies