"What If Smart Homes Could See Our Homes?": Exploring DIY Smart Home Building Experiences with VLM-Based Camera Sensors

2025-03-04

AI Summary

Visual language models (VLMs) promise to transform DIY smart home development, yet their impact on end-user programming practices, usability challenges, and privacy perceptions remains underexplored. Method: We conducted a three-week diary study with participants iteratively using a VLM-powered camera prototype (e.g., GPT-4V) to autonomously interpret home scenes, define requirements, customize logic, and test smart functions—without coding. Contribution/Results: This work provides the first empirical characterization of user expectations (e.g., semantic interaction, zero-shot generalization), identifies key tensions in the DIY workflow (e.g., lowered programming barriers versus novel debugging complexity), and surfaces critical concerns around privacy and controllability. Based on these findings, we propose three human-centered design principles: progressive semantic authorization, explainable behavioral feedback, and context-aware privacy boundaries. Our study establishes the first evidence-based foundation and actionable design pathway for VLM-driven, end-user programmable smart homes.

Technology Category

• Computer Vision: Language and Vision
• Humans and AI: Intelligent User Interfaces
• Intelligent Robots: Human-Robot Interaction

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

The advancement of Vision-Language Model (VLM) camera sensors, which enable autonomous understanding of household situations without user intervention, has the potential to completely transform the DIY smart home building experience. Will this simplify or complicate the DIY smart home process? Additionally, what features do users want to create using these sensors? To explore this, we conducted a three-week diary-based experience prototyping study with 12 participants. Participants recorded their daily activities, used GPT to analyze the images, and manually customized and tested smart home features based on the analysis. The study revealed three key findings: (1) participants' expectations for VLM camera-based smart homes, (2) the impact of VLM camera sensor characteristics on the DIY process, and (3) users' concerns. Through the findings of this study, we propose design implications to support the DIY smart home building process with VLM camera sensors, and discuss living with intelligence.

Problem

• Exploring DIY smart home experiences with VLM camera sensors.
• Investigating user expectations and concerns for VLM-based smart homes.
• Proposing design implications for DIY smart home building using VLM sensors.

Innovation

• VLM camera sensors for autonomous household understanding
• Diary-based prototyping with GPT image analysis
• Customizable smart home features using VLM sensors