AutoCue: Multimodal LLM-Assisted Externalization of Implicit Inputs as Instructional Visual Cues in Screencast Tutorials

2026-08-05

AI Summary

This study addresses the frequent disruptions learners experience when watching screen-recorded software tutorials, often caused by missing implicit inputs such as mouse actions and keyboard modifier keys. To mitigate this issue, the authors propose AutoCue—a novel approach that combines multimodal large language models with human-in-the-loop refinement to infer and explicitly visualize hidden user interactions. By analyzing visual frame transitions, spoken narration, and official documentation, AutoCue introduces a first-of-its-kind visual cue grammar tailored for software learning, generating editable overlay prompts that effectively reduce cognitive load. Experimental results on Maya tutorials demonstrate that AutoCue significantly decreases task completion time, minimizes interaction breakdowns, and enhances overall learning experience.

Abstract

Tutorial videos are widely used for learning feature-rich software, yet following screencast tutorials often breaks down in practice. Through a survey and contextual inquiry, we found that learners frequently rewind or get stuck because critical input information, especially mouse actions and keyboard-modified operations, is often implicit or missing in tutorials without input metadata. To address this problem, we present AutoCue, a multimodal LLM-assisted, human-in-the-loop tutorial augmentation pipeline for externalizing implicit inputs as instructional visual cues. AutoCue integrates frame-to-frame visual changes, narration signals, and operation guidance from official software manuals to infer likely mouse and key-modifier actions, then produces aligned cue layers and editable artifacts for human refinement. Grounded in multimedia learning and cognitive load theory, we further develop a visual cue grammar for representing mouse, keyboard, and combined inputs in software-learning tutorials. We instantiate and evaluate AutoCue in Autodesk Maya, focusing automatic inference on selected UI-mediated interactions with observable visual or textual feedback while supporting more ambiguous state changes through editable authoring artifacts. In a between-subjects study with 24 participants, the AutoCue-augmented tutorial reduced task completion time and interaction breakdowns and showed improved learner-reported experience.

Problem

• screencast tutorials
• implicit inputs
• visual cues
• tutorial comprehension
• input externalization

Innovation

• multimodal LLM
• implicit input externalization
• visual cue grammar
• human-in-the-loop augmentation
• screencast tutorial enhancement