PromptShield Home: Ambient Multimodal Prompt Injection Defense for Smart-Home Agents

2026-08-05

AI Summary

This work addresses the vulnerability of smart home agents to multimodal prompt injection attacks—such as voice from televisions, on-screen text, or overheard conversations—that can trigger unintended actions. It presents the first systematic characterization of such threats in domestic environments and introduces PromptShield-Home, a benchmark encompassing representative scenarios including ambiguous addressing, audiovisual injection, and spurious health-monitoring triggers. The study evaluates defense efficacy across three abstraction levels: conventional detectors (L0), monolithic multimodal large models (L1), and multi-agent collaborative mechanisms (L2). Experiments reveal that even the best single-method approach achieves only a 76.5% safe execution rate, whereas an ideal routing strategy raises this to 94.1%. Notably, detectors tend toward over-execution while multimodal models frequently reject legitimate commands, exhibiting complementary failure modes that underscore the necessity of integrating learned routing with multi-sensor collaboration.

Abstract

Smart-home assistants increasingly use multimodal large language models (MLLMs) that perceive video and audio directly. This raises a safety question specific to the home: can the agent tell a genuine user command from ambient or externally-sourced content, television speech, on-screen text, or an overheard conversation, that merely looks like a command? We introduce PromptShield-Home, a pilot benchmark of realistic smart-home scenarios spanning addressee ambiguity, screen/audio injection, health-monitor false triggers, mixed occupancy, and a legitimate-command floor, and use it to compare three abstraction layers: traditional detectors (L0), a single MLLM agent (L1; vision, vision+ASR, and audio-visual), and multi-agent mediation (L2; voting, role specialists, cross-model arbitration). Because the label distribution is skewed toward inaction, aggregate accuracy is misleading, a constant always-block predictor scores 82%, so we report unsafe-execution and safe-completion rates separately. The two paradigms fail in opposite ways: detectors act on everything, while every MLLM configuration over-refuses, completing almost no genuine command and missing a true fall in every case. Crucially, their correct sets are disjoint: an oracle that always picks the right layer reaches 94.1%, against 76.5% for the best single layer. We report this as an upper bound, not a system - no router is implemented - and argue that home-agent safety is best served by learned routing and sensor fusion, not by replacing detectors with an MLLM.

Problem

• multimodal prompt injection
• smart-home agents
• command authenticity
• ambient content
• safety

Innovation

• multimodal prompt injection
• smart-home safety
• MLLM defense
• multi-agent mediation
• sensor fusion