Adobe Summit Concierge Evaluation with Human in the Loop

2025-11-05

AI Summary

This work addresses the practical challenges of deploying enterprise-grade generative AI assistants under conditions of sparse domain data, stringent quality assurance requirements, and urgent time-to-deployment constraints. Focusing on Adobe Summit as a representative use case, we design and implement Summit Concierge—a domain-specific AI assistant. Our approach introduces a human-in-the-loop development paradigm that integrates lightweight prompt engineering, retrieval-augmented generation (RAG), and iterative human validation to enable agile cold-start iteration and controlled production deployment. The system supports real-time knowledge updates, multi-turn context-aware meeting queries, and an embedded quality feedback loop. Empirical evaluation demonstrates significant improvements in response accuracy and user satisfaction, validating the scalability, reliability, and engineering feasibility of human-AI collaborative development for enterprise AI assistants.

Technology Category

• Humans and AI: AI for Accessibility
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Natural Language Processing: Prompt Engineering / Prompting

Application Category

• Search and Retrieval-Augmented AI: Agentic search
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

Generative AI assistants offer significant potential to enhance productivity, streamline information access, and improve user experience in enterprise contexts. In this work, we present Summit Concierge, a domain-specific AI assistant developed for Adobe Summit. The assistant handles a wide range of event-related queries and operates under real-world constraints such as data sparsity, quality assurance, and rapid deployment. To address these challenges, we adopt a human-in-the-loop development workflow that combines prompt engineering, retrieval grounding, and lightweight human validation. We describe the system architecture, development process, and real-world deployment outcomes. Our experience shows that agile, feedback-driven development enables scalable and reliable AI assistants, even in cold-start scenarios.

Problem

• Develops domain-specific AI assistant for enterprise events
• Addresses data sparsity and quality assurance challenges
• Implements human-in-the-loop workflow for reliable deployment

Innovation

• Human-in-the-loop development workflow
• Prompt engineering with retrieval grounding
• Lightweight human validation for quality