MarkupLens: An AI-Powered Tool to Support Designers in Video-Based Analysis at Scale

2024-03-08

arXiv.org

AI Summary

Current video-based design (VBD) workflows rely on inefficient and inconsistent manual video annotation, severely limiting the efficiency and quality of interaction insight generation. To address this, we propose MarkupLens—an AI-augmented video annotation platform featuring a novel three-tier progressive AI-assisted annotation mechanism that optimizes human–AI collaboration while preserving designer agency. MarkupLens integrates state-of-the-art visual understanding models, eye-tracking data, and a human–AI collaboration evaluation framework to deliver a lightweight interactive prototype. A user study with 36 professional designers demonstrates a 37% increase in annotation efficiency, a 29% improvement in inter-annotator consistency, a 41% reduction in subjective cognitive load, and significantly higher UX satisfaction. Critically, this work provides the first empirical eye-tracking validation that AI-assisted annotation can simultaneously achieve automation efficacy and human-centered control—establishing a new paradigm for trustworthy, controllable AI support in VBD.

Technology Category

• Computer Vision: Video Understanding & Activity Analysis
• Humans and AI: Game Design — Virtual Humans, NPCs and Autonomous Characters
• Intelligent Robots: Human-Robot Interaction

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Video-Based Design (VBD) is a design methodology that utilizes video as a primary tool for understanding user interactions, prototyping, and conducting research to enhance the design process. Artificial Intelligence (AI) can be instrumental in video-based design by analyzing and interpreting visual data from videos to enhance user interaction, automate design processes, and improve product functionality. In this study, we explore how AI can enhance professional video-based design with a State-of-the-Art (SOTA) deep learning model. We developed a prototype annotation platform (MarkupLens) and conducted a between-subjects eye-tracking study with 36 designers, annotating videos with three levels of AI assistance. Our findings indicate that MarkupLens improved design annotation quality and productivity. Additionally, it reduced the cognitive load that designers exhibited and enhanced their User Experience (UX). We believe that designer-AI collaboration can greatly enhance the process of eliciting insights in video-based design.

Problem

• Addressing labor-intensive manual video annotation in Video-Based Design workflows
• Investigating how Computer Vision automation levels affect annotation quality and cognitive load
• Exploring human-AI collaboration trade-offs between automation assistance and user control

Innovation

• CV-assisted platform balances automation and control
• Three-tiered support system improves annotation quality
• Adjustable autonomy reduces cognitive load in annotation