Piloting Planetarium Visualizations with LLMs during Live Events in Science Centers

2026-01-28

AI Summary

This work proposes and implements the first large language model (LLM)-based AI co-pilot system integrated into the OpenSpace astronomical visualization platform to address the heavy workload and multitasking coordination challenges faced by human operators in traditional planetarium shows. By leveraging real-time speech recognition and natural language command parsing, the system autonomously executes operations such as camera movement, time simulation, and visual asset switching. This study represents the first application of LLMs to real-time planetarium control, exploring AI’s role as a collaborative co-pilot alongside human presenters. In evaluations involving seven experienced planetarium narrators, the system—while not yet fully autonomous—demonstrated significant potential to reduce operational burden, facilitate parallel task management, and enhance human-AI collaboration.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

We designed and evaluated an AI pilot in a planetarium visualization software, OpenSpace, for public shows in science centers. The piloting role is usually given to a human working in close collaboration with the guide on stage. We recruited 7 professional guides with extensive experience in giving shows to the public to study the impact of the AI-piloting on the overall experience. The AI-pilot is a conversational AI-agent listening to the guide and interpreting the verbal statements as commands to execute camera motions, change simulation time, or toggle visual assets. Our results show that, while AI pilots lack several critical skills for live shows, they could become useful as co-pilots to reduce workload of human pilots and allow multitasking. We propose research directions toward implementing visualization pilots and co-pilots in live settings.

Problem

• planetarium visualization
• AI pilot
• live science shows
• conversational AI
• human-AI collaboration

Innovation

• LLM-based AI agent
• planetarium visualization
• live event interaction
• human-AI collaboration
• conversational command interpretation