Problem
Research questions and friction points this paper is trying to address.
Multi-user interaction
3D virtual environments
Addressee identification
Speaker orientation
Privacy oversharing
Who Is Talking to the Agent? LLMs in Multi-User 3D Virtual Environments
🤖 AI Summary
This study addresses recipient identification and privacy leakage issues for LLM-based agents in multi-user 3D virtual environments. We construct the LookAway corpus, leveraging multimodal large models and 3D simulation to investigate how spatial orientation influences intent determination and how user profiles introduce privacy risks. Our findings reveal that visual cues readily mislead model judgments; although accuracy reaches 99.5% when orientations align, severe biases persist. Furthermore, while providing user profiles enhances response quality, it results in excessive privacy disclosure in 45.3% of cases. This work quantifies the trade-off between contextual richness and privacy preservation, offering critical insights for ensuring secure agent interactions within virtual environments.
📝 Abstract
When several people share a 3D virtual room with an LLM agent, the agent must decide not only what to say, but whether an utterance was addressed to it and, if accessible, what profile information about the others present it may use. To study both problems, we construct LookAway, a controlled corpus of 40 sessions involving 80 distinct personas and an LLM agent (1,200 turns), including ambiguous-addressee turns in which speaker orientation agrees or conflicts with the intended addressee. Across three open-weight large language models and five conditions varying which profiles the agent sees and whether it is told where each person faces (18,000 decisions), adding speaker orientation increased addressee accuracy from 56% to 99.5% when orientation was congruent, but when the speaker faced someone other than the addressee, two of the models went by where the speaker faced on more than 85% of those turns. Warning one model that orientation could be misleading reduced this only modestly. A browser-based 3D demonstrator shows the effect live: the same sentence gets an answer when the speaker faces the agent and silence when they face the other person. Providing both personas' profiles improved responses about the person being asked about, but also increased the use of profile attributes not revealed in the shared conversation, reaching 45.3% of answers for one model. More context thus improves multi-user interaction but also leads to oversharing, so shared LLM agents need mechanisms for weighing spatial cues and controlling when user-specific information enters a response.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Multi-user 3D virtual environments
Addressee detection
Speaker orientation
User persona oversharing
LLM agents
🔎 Similar Papers
No similar papers found.
