AFA: Identity-Aware Memory for Preventing Persona Confusion in Multi-User Dialogue

2026-04-27

AI Summary

This work addresses the challenges of personalized information leakage and role confusion that arise when multiple users share a voice assistant due to historical context mixing. To mitigate these issues, the authors propose the Adaptive Friend Agent (AFA) framework, which leverages speaker verification and user-specific memory banks to enable identity-aware personalized dialogue. The study formally defines and quantifies the "role confusion" problem for the first time, introduces an identity-aware routing mechanism, and constructs PAT—a synthetic dataset comprising 133 user personas—along with the accompanying evaluation protocol PAA. Experimental results demonstrate that AFA improves performance on the PAA metric from 35.7% to 61.3%, while human evaluations confirm a significant enhancement in response personalization, thereby validating the critical role of identity-aware routing in alleviating role confusion.

Abstract

When multiple people share a single voice assistant, the system conflates their histories: one resident's preferences can leak into another's responses, eroding utility and trust. We call this failure mode persona confusion, and we show it is a measurable problem in today's single-user dialogue systems when deployed in shared environments. We present the Adaptive Friend Agent (AFA), a modular framework that combines voice-based speaker identification with per-user memory stores to enable identity-aware, personalized dialogue across multiple users. To support training and evaluation, we construct PAT (Personalized Agent chaT), a synthetic dataset of 58,289 persona-grounded dialogue turns spanning 133 user profiles and 12 real-world scenarios. We evaluate AFA across five LLM back-ends in a standard response-quality benchmark, with a LLaMA-2-70B model fine-tuned on PAT achieving the highest overall performance. To directly measure persona confusion prevention, we introduce an interleaved multi-user evaluation protocol with a novel metric, Persona Attribution Accuracy (PAA), demonstrating that identity-aware routing improves PAA from 35.7% to 61.3%. Human evaluation confirms annotators perceive significantly higher personalization in routing-enabled responses. Our results establish that identity-aware user routing is the critical component for preventing persona confusion in multi-user conversational systems.

Problem

• persona confusion
• multi-user dialogue
• voice assistant
• identity-awareness
• personalization

Innovation

• identity-aware memory
• persona confusion
• multi-user dialogue
• speaker identification
• personalized agent