Selective Attention System (SAS): Device-Addressed Speech Detection for Real-Time On-Device Voice AI

2026-04-09

AI Summary

This work addresses the challenge of device-directed speech detection in edge-device scenarios characterized by multiple concurrent speakers and temporally ambiguous speech. The authors propose a Selective Attention System (SAS) that formalizes the task as Sequence-based Device-Directed Attention Routing (SDAR), leveraging short-term interaction history for contextual decision-making. SAS introduces, for the first time, a causal interaction history modeling framework combined with a lightweight audio-visual fusion mechanism, enabling real-time on-device inference on ARM Cortex-A class hardware. Experimental results demonstrate that the audio-only model achieves an F1 score of 0.86, which improves to 0.95 upon incorporating visual cues. The system operates with end-to-end latency under 150 ms and consumes less than 20 MB of memory, effectively balancing computational efficiency with high accuracy.

Technology Category

• Machine Learning: Hardware-aware ML
• Natural Language Processing: Speech
• Humans and AI: Interaction Techniques and Devices

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

We study device-addressed speech detection under pre-ASR edge deployment constraints, where systems must decide whether to forward audio before transcription under strict latency and compute limits. We show that, in multi-speaker environments with temporally ambiguous utterances, this task is more effectively modelled as a sequential routing problem over interaction history than as an utterance-local classification task. We formalize this as Sequential Device-Addressed Routing (SDAR) and present the Selective Attention System (SAS), an on-device implementation that instantiates this formulation. On a held-out 60-hour multi-speaker English test set, the primary audio-only configuration achieves F1=0.86 (precision=0.89, recall=0.83); with an optional camera, audio+video fusion raises F1 to 0.95 (precision=0.97, recall=0.93). Removing causal interaction history (Stage~3) reduced F1 from 0.95 to 0.57+/-0.03 in the audio+video configuration under our evaluation protocol. Among the tested components, this was the largest observed ablation effect, indicating that short-horizon interaction history carries substantial decision-relevant information in the evaluated setting. SAS runs fully on-device on ARM Cortex-A class hardware (<150 ms latency, <20 MB footprint). All results are from internal evaluation on a proprietary dataset evaluated primarily in English; a 5-hour evaluation subset may be shared for independent verification (Section 8.8).

Problem

• device-addressed speech detection
• on-device voice AI
• multi-speaker environments
• real-time speech routing
• pre-ASR decision

Innovation

• Selective Attention System
• Sequential Device-Addressed Routing
• on-device voice AI
• interaction history modeling
• audio-visual fusion