OmniSmartHome: A Multimodal Reasoning Benchmark for Smart-Home Agents

2026-09-26

AI Summary

This study addresses the challenge that smart home assistants struggle to process multimodal ambiguous requests relying on visual and spatial audio contexts. To this end, it introduces the first smart home benchmark integrating multimodal context across both synthetic and real-world scenarios. Methodologically, the authors propose PROME, a procedural memory-based agent framework that synergizes omni-modal large language models with audiovisual perception tools and memory mechanisms to optimize evidence gathering and reasoning. Experimental results reveal significant deficiencies in existing models regarding multimodal reasoning, while demonstrating that PROME effectively enhances the execution performance of six mainstream omni-modal large language models.

Abstract

Smart-home assistants are expected to handle diverse, realistic requests that arise in daily life. In such interactions, users often rely on the surrounding multimodal context-pointing at objects or referring to what they see or hear, leaving their requests underspecified in language alone. Existing smart-home benchmarks, however, express user requests solely through language, leaving context-dependent real-world requests underexplored. To bridge this gap, we introduce OmniSmartHome, a multimodal smart-home benchmark where each spoken request is paired with the surrounding visual and spatial-audio context, providing complementary cues to disambiguate underspecified requests. OmniSmartHome comprises 1,360 synthetic and 272 real-world episodes. We evaluate 16 omnimodal large language models (Omni-LLMs) and reveal that, while they perform strongly when speech alone sufficiently conveys the user's intent, performance drops substantially when resolving it requires reasoning over multimodal contextual cues. As a simple agent baseline, we provide PROME (PROcedural Memory for multimodal Evidence gathering), which equips agents with specialized audio-visual perception tools and procedural memory for orchestrating their use. PROME generally improves performance across six Omni-LLMs. Demos and examples are available at https://omni-smart-home.github.io

Problem

• smart-home agents
• multimodal reasoning
• benchmark
• underspecified requests
• omnimodal large language models

Innovation

• Multimodal Reasoning
• Smart-Home Agents
• Omni-LLMs
• Procedural Memory
• Spatial-Audio Context