Ego-METAS: Egocentric online Multimodal Energy-efficient Temporal Action Segmentation benchmark

2026-05-29

AI Summary

This work addresses the challenge of enabling resource-constrained embodied agents to efficiently perceive and accurately segment temporal actions under limited energy budgets during continuous operation. To this end, we introduce the first egocentric, online, multimodal, and energy-aware benchmark for temporal action segmentation. The benchmark integrates five modalities—RGB, audio, eye-tracking, IMU, and monochrome video—and incorporates a hardware-mappable energy budgeting mechanism alongside a dynamic sensor routing strategy. It also provides standardized data splits, annotations, and pre-extracted features. Experimental results demonstrate that optimal sensor selection is highly context-dependent, that existing methods struggle to adapt to continuously varying environments, and that even simple dynamic multimodal fusion can substantially improve the trade-off between energy efficiency and segmentation accuracy.

Abstract

To operate in the physical world, embodied agents must perceive their environment in an "always-on" fashion, selectively accessing the most informative sensors to balance energy constraints and task accuracy. Despite its importance for resource-constrained devices, energy-aware perception remains under-explored, with most prior work assuming unlimited compute. To address this, we introduce Ego-METAS: the first Egocentric online Multimodal Energy-efficient Temporal Action Segmentation benchmark. Ego-METAS provides a unified testbed of more than 100 hours of untrimmed egocentric video from EgoExo4D, CMU-MMAC, and CaptainCook4D, spanning 5 modalities (RGB, audio, gaze, IMU, and monochrome camera). We formulate an online temporal action segmentation task where models must dynamically select which sensors to activate at each timestep while strictly adhering to hardware-representative energy budgets. Alongside the benchmark, we release unified splits, cleaned annotations, pre-extracted features, and a diverse suite of baseline routing policies. Our evaluations show that optimal routing is highly scenario-dependent, and that existing policy-learning methods, designed primarily for trimmed clips, struggle to adapt to continuous, untrimmed environments. However, even simple dynamic fusion of complementary modalities (e.g., via random routing) proves critical for balancing predictive accuracy against strict energy budgets. Ultimately, Ego-METAS provides a standardized foundation to develop robust, cost-aware policies for autonomous, always-on embodied AI.

Problem

• energy-efficient perception
• temporal action segmentation
• multimodal sensing
• embodied AI
• online perception

Innovation

• energy-efficient perception
• online temporal action segmentation
• multimodal sensor routing
• egocentric vision
• embodied AI