Problem
Research questions and friction points this paper is trying to address.
Augmented Reality Captioning
Speaker Attribution
Speaker Diarization
Speaker Recognition
In-Conversation Registration
VocalEyes: Speaker-Aware Augmented Reality Captioning through In-Conversation Registration
🤖 AI Summary
This study addresses the limitation of conventional AR captions in distinguishing speaker identities, which impedes users’ ability to follow multi-party conversations. To overcome this, we propose a speaker-aware AR captioning system that leverages natural self-introductions. The system introduces a novel in-conversation enrollment mechanism that eliminates the need for prior registration by fusing acoustic and visual cues to construct named voiceprint profiles. Combined with a multimodal interface, it enables real-time speaker attribution. Experimental evaluations demonstrate that the proposed system achieves an 88% speaker identification accuracy and improves users’ conversation tracking accuracy to 87.3%, while significantly reducing cognitive load.
📝 Abstract
Co-located augmented reality (AR) captions make speech readable, but they can separate an utterance from the person who produced it. In unfamiliar groups, losing that source complicates immediate responses and later review: users must recover not only what was said, but also who said it. Conventional diarization returns anonymous clusters, while speaker recognition typically assumes pre-meeting enrollment. We built VocalEyes, a speaker-aware AR captioning system that creates named voice profiles from natural self-introductions. The interface coordinates speaker-attributed captions, a fixed profile card, and a visual cue that marks the articulating face. In a controlled within-subjects study with 20 participants who reported typical hearing, VocalEyes identified speakers with 88.0% accuracy and increased participant speaker-tracking accuracy from 47.2% with caption-only AR to 87.3% with the complete interface. Participants also reported lower workload with the complete interface. These findings show how in-conversation registration can preserve speaker attribution across live captions and meeting records in scripted small-group meetings.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Augmented Reality Captioning
Speaker-Aware
In-Conversation Registration
Voice Profiles
Speaker Diarization
🔎 Similar Papers
No similar papers found.
