MaRginalia: Enabling In-person Lecture Capturing and Note-taking Through Mixed Reality

2025-01-27

AI Summary

Students face challenges—including low note-taking efficiency, increased distraction, and omission of critical content—when using electronic devices for in-class note-taking. To address these issues, this paper introduces a mixed reality (MR)-native note-taking system implemented on the HoloLens 2. The system enables gaze-driven, head-up interaction without downward glances, integrating spatially anchored slide rendering, real-time speech-to-text transcription, and stylus-based handwriting capture. It further supports seamless dual-mode collaboration between physical tablets and virtual MR interfaces. This work establishes the first MR-native note-taking paradigm, underpinned by fused eye-gesture interaction and multimodal temporal alignment to ensure precise information synchronization. A user study (N=12) demonstrates that the system improves note capture speed by 3.2×, significantly enhances note completeness and classroom focus, and achieves 92% participant willingness to adopt it in authentic instructional settings.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Machine Learning: Hardware-aware ML

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web applications in cross-disciplinary domains and verticals such as mixed reality, smart cities, and digital health
• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG

Abstract

Students often take digital notes during live lectures, but current methods can be slow when capturing information from lecture slides or the instructor's speech, and require them to focus on their devices, leading to distractions and missing important details. This paper explores supporting live lecture note-taking with mixed reality (MR) to quickly capture lecture information and take notes while staying engaged with the lecture. A survey and interviews with university students revealed common note-taking behaviors and challenges to inform the design. We present MaRginalia to provide digital note-taking with a stylus tablet and MR headset. Students can take notes with an MR representation of the tablet, lecture slides, and audio transcript without looking down at their device. When preferred, students can also perform detailed interactions by looking at the physical tablet. We demonstrate the feasibility and usefulness of MaRginalia and MR-based note-taking in a user study with 12 students.

Problem

• Note-taking
• Distraction
• Information Loss

Innovation

• Mixed Reality
• Note-taking System
• Attention Management