FlexiFly: Interfacing the Physical World with Foundation Models Empowered by Reconfigurable Drone Systems

2024-03-19

AI Summary

Current foundation models face a “broad-area perception–localized event” mismatch in physical-world sensing and interaction: dense sensor deployment is required to cover large spaces, yet precise localization of critical events in small regions remains challenging. To address this, we propose FlexiFly—a novel platform featuring (1) a semantics-driven image segmentation method enabling foundation models to actively identify task-relevant regions; (2) a modular, reconfigurable drone architecture integrating sensing and actuation, supporting real-time large-model scheduling and adaptive payload switching; and (3) an end-to-end stack comprising semantic-guided segmentation, multimodal instruction parsing, autonomous navigation, and edge-cloud collaborative inference. Evaluated in real-world smart home environments, FlexiFly achieves an 85% task completion rate—demonstrating, for the first time, the feasibility and effectiveness of transitioning foundation models from digital content generation to embodied physical interaction.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Machine Learning: Deep Neural Architectures and Foundation Models

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Search and Retrieval-Augmented AI: Agentic search
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

Foundation models (FM) have shown immense human-like capabilities for generating digital media. However, foundation models that can freely sense, interact, and actuate the physical domain is far from being realized. This is due to 1) requiring dense deployments of sensors to fully cover and analyze large spaces, while 2) events often being localized to small areas, making it difficult for FMs to pinpoint relevant areas of interest relevant to the current task. We propose FlexiFly, a platform that enables FMs to ``zoom in'' and analyze relevant areas with higher granularity to better understand the physical environment and carry out tasks. FlexiFly accomplishes by introducing 1) a novel image segmentation technique that aids in identifying relevant locations and 2) a modular and reconfigurable sensing and actuation drone platform that FMs can actuate to ``zoom in'' with relevant sensors and actuators. We demonstrate through real smart home deployments that FlexiFly enables FMs and LLMs to complete diverse tasks up to $85%$ more successfully. FlexiFly is critical step towards FMs and LLMs that can naturally interface with the physical world.

Problem

• Enabling foundation models to sense and interact with physical environments.
• Addressing challenges in pinpointing relevant areas of interest in large spaces.
• Developing a reconfigurable drone platform for enhanced environmental analysis.

Innovation

• Reconfigurable drone system for physical interaction
• Novel image segmentation for location identification
• Modular sensing and actuation for granular analysis