Metabook: A System to Automatically Generate Interactive AR Storybooks to Improve Children's Reading
🤖 AI Summary
To address low reading engagement among children, high production costs of AR content, and limited interactivity, this work proposes the first end-to-end framework for automatically generating interactive AR-enabled 3D storybooks. Methodologically, it integrates large language models (for story comprehension and storyboard script generation), multimodal generative models (for cross-modal translation from 2D illustrations to 3D assets), and a real-time AR rendering and embodied interaction system powered by SLAM and Unity/ARKit/ARCore. The resulting 3D reading companion supports real-time question-answering and cognitive co-reading. Empirical evaluation demonstrates significant improvements in children’s reading engagement, vocabulary retention, and narrative comprehension; educators strongly endorse its pedagogical value in integrating literacy skills and strengthening language–visual thinking connections. This work pioneers fully automated, low-barrier generation of semantically rich, interactive AR storybooks directly from textual input.
Application Category
📝 Abstract
Reading is important for children to acquire knowledge, enhance cognitive abilities, and improve language skills. However, current reading methods either offer limited visual presentation, making them less interesting to children, or lack channels for children to share insights and ask questions during reading. AR/VR books provide rich visual cues that address the issue of children's lack of interest in reading, but the high production costs and need for professional expertise limit the volume of AR/VR books and children's choices. We propose Metabook, a system to automatically generate interactive AR storybooks to improve children's reading. Metabook introduces a story-to-3D-book generation scheme and a 3D avatar that combines multiple AI models as a reading companion. We invited six primary and secondary school teachers to conduct a formative study to explore the design considerations for an ideal children's AR reading tool. In the user study, we invited relevant professionals (art, computer science professionals, and a semanticist), 44 children, and six teachers to evaluate Metabook. Our user study shows that Metabook can significantly increase children's interest in reading and deepen their impression of reading materials and vocabulary in books. Teachers acknowledged Metabook's effectiveness in facilitating reading communication and enhancing reading enthusiasm by connecting verbal and visual thinking, expressing high expectations for its future potential in education.
Problem
Research questions and friction points this paper is trying to address.
Automating 3D book creation for AR to reduce time and skill barriers
Enabling novice users to generate 3D books from text effortlessly
Evaluating AR 3D books' impact on children's learning outcomes
Innovation
Methods, ideas, or system contributions that make the work stand out.
AI automates 3D book creation from text
Mobile-to-headset pipeline for AR books
End-to-end system for novice users
🔎 Similar Papers
