Demo2Tutorial: From Human Experience to Multimodal Software Tutorials

2026-06-02

AI Summary

This study addresses the challenge of automatically extracting structured, reusable software tutorial knowledge from raw user interactions in digital environments. To this end, we propose an end-to-end framework that leverages screen recordings and interaction logs to generate high-quality multimodal tutorials directly from unedited human demonstrations. By integrating multimodal action parsing, hierarchical task graph planning, and tutorial synthesis, our approach is the first to autonomously produce instructional content from real-world user behavior. The resulting tutorials not only surpass both manually authored guides and existing baselines in quality but also significantly enhance human learning efficiency. Furthermore, they effectively improve the task planning and generalization capabilities of GUI-based intelligent agents.

Abstract

Human experience in digital environments offers a vast, underexplored resource of authentic, untrimmed interactions that contain rich procedural knowledge. We introduce Demo2Tutorial, a framework that transforms this experience captured via screen recordings and interaction logs into structured, multimodal software tutorials for teaching both humans and agents. Demo2Tutorial first collects human experience via a dedicated recorder, then parses raw experience using a multimodal Action Parser to reconstruct perception, action, and intent. A Step Planner then abstracts these steps into hierarchical task graphs representing goals and steps. Finally, a Tutorial Composer transforms the parsed experience into structured, reusable image-text instructions. We evaluate the tutorial generation quality on a new benchmark derived from official software documentation. We further demonstrate that this distilled representation benefits (i) human learning, by automatically generating multimodal tutorials, and (ii) agent learning, by improving downstream GUI-agent planning and generalization. Experiments show Demo2Tutorial produces high-quality tutorials that surpass human-authored ones and significantly outperform baseline methods, while enabling both faster human task completion and improved GUI agent planning, demonstrating that structured tutorials distilled from human experience can serve as effective knowledge representations for advancing both human learning and agent capabilities. Code and data will be available at https://github.com/showlab/Demo2Tutorial.

Problem

• human experience
• multimodal tutorials
• software documentation
• procedural knowledge
• GUI agents

Innovation

• multimodal tutorial generation
• human experience distillation
• hierarchical task graph
• GUI agent learning
• action parsing