SPICE: Smart Projection Interface for Cooking Enhancement

2024-12-04

arXiv.org

AI Summary

Existing HCI research insufficiently explores tangible user interfaces (TUIs) for bimanual, real-world tasks—such as cooking—and lacks effective methods to transform textual recipes into spatialized, context-aware interactive experiences. This paper introduces SPICE, the first systematic TUI framework designed specifically for everyday bimanual physical manipulation. SPICE integrates real-time object tracking, agent-driven kitchen simulation, and a vision-language large model (VLLM) to jointly enable environment perception and semantic recipe understanding; it dynamically projects step-by-step instructions onto physical cooking surfaces, creating an accessible, interactive tangible interface. A controlled study with 30 participants demonstrates that SPICE significantly reduces task completion time (p < 0.01) and minimizes procedural interruptions (p < 0.001), validating its efficacy in enhancing interaction fluency and task continuity. The work establishes a novel paradigm for deploying TUIs in complex, dynamic physical environments.

Technology Category

• Humans and AI: Interaction Techniques and Devices
• Intelligent Robots: Manipulation
• Cognitive Modeling & Cognitive Systems: Social Cognition And Interaction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

Tangible User Interfaces (TUI) for human--computer interaction (HCI) provide the user with physical representations of digital information with the aim to overcome the limitations of screen-based interfaces. Although many compelling demonstrations of TUIs exist in the literature, there is a lack of research on TUIs intended for daily two-handed tasks and processes, such as cooking. In response to this gap, we propose SPICE (Smart Projection Interface for Cooking Enhancement). SPICE investigates TUIs in a kitchen setting, aiming to transform the recipe following experience from simply text-based to tangibly interactive. SPICE uses a tracking system, an agent-based simulation software, and vision large language models to create and interpret a kitchen environment where recipe information is projected directly onto the cooking surface. We conducted comparative usability and a validation studies of SPICE, with 30 participants. The results show that participants using SPICE completed the recipe with far less stops and in a substantially shorter time. Despite this, participants self-reported negligible change in feelings of difficulty, which is a direction for future research. Overall, the SPICE project demonstrates the potential of using TUIs to improve everyday activities, paving the way for future research in HCI and new computing interfaces.

Problem

• Lack of TUIs for daily two-handed tasks like cooking
• Transforming text-based recipes into tangible interactive experiences
• Improving cooking efficiency with projected recipe interfaces

Innovation

• Tracking system for kitchen environment interaction
• Agent-based simulation software for recipe projection
• Vision large language models for environment interpretation