BotDirector: Robot Storytelling Across the Symmetrical Reality with Multi-modal Interactions

2026-06-02

AI Summary

This work addresses the high technical barriers that hinder children’s participation in robot-based storytelling. The authors propose a low-threshold creative system integrating tangible interaction, natural language, and swarm robotics, enabling children to co-create stories by arranging everyday objects in collaboration with a large language model. The system automatically maps these physical arrangements into executable action sequences, which drive autonomously navigating swarm robots to perform theatrical enactments. By innovatively combining multimodal interaction with a scene-to-action mapping algorithm, the approach significantly enhances children’s engagement and creativity, allowing them to flexibly author and instantly visualize personalized robotic narratives.

Abstract

Robot storytelling offers a unique blend of technological innovation and creative expression that engages children in unprecedented ways. However, the technical aspects are often too complicated for children. We propose an interactive system that facilitates robot storytelling with tangible and natural language interactions. Children arrange the playground with their own stuff and create narratives with an LLM agent. The created narratives are transformed into a motion sequence based on the map and characters, and the motions are executed by self-navigating swarm robots. This system enhances robot storytelling with flexible scenarios, enabling young children to create robot dramas with everyday objects.

Problem

• robot storytelling
• child-friendly interaction
• multi-modal interaction
• tangible interface
• symmetrical reality

Innovation

• robot storytelling
• tangible interaction
• natural language interaction
• swarm robots
• large language model