NextMe-800: Anticipating Personal Behavior from Months of Egocentric Video

2026-10-01

AI Summary

This study addresses the challenge of predicting future events from personal historical behavior to optimize decision-making by proposing an end-to-end solution. Methodologically, it introduces EgoLife, an 800-hour first-person video dataset with accompanying benchmarks, a multi-granularity annotation scheme, and an open-vocabulary sequence prediction framework, alongside a soft edit distance metric for model evaluation. The primary contributions lie in providing behavioral assessment resources spanning multiple abstraction levels and prediction horizons, effectively supporting long-horizon personalized action forecasting. Furthermore, this work empirically validates the predictability of long-term individual behavior, offering a comprehensive foundation for advancing research in egocentric anticipation and personalized predictive modeling.

Abstract

We often plan ambitiously yet act habitually and wonder, in retrospect, whether we would have planned differently had we known what we would actually do. Hindsight offers a valuable perspective on past decisions, although we often wish we could have simulated hindsight at the moment of choosing. If a system could generate plausible trajectories from one's personal history, such previews might help people formulate more realistic plans and make better informed decisions. We introduce NextMe-800, an approximately 800-hour first-person dataset from one volunteer over 126 days with 1 Hz images, gaze, and audio, captioned at five hierarchical abstraction levels from atomic actions to major activities. We formulate personalized action anticipation as open-vocabulary K-step sequence prediction and construct NextAct, a 1,500-point benchmark combining NextMe-800 with the multi-person EgoLife dataset. Using an embedding-based soft edit distance as the metric, we evaluate how well different models can anticipate personal behavior across abstraction levels and prediction horizons. NextMe-800 and NextAct provide a months-long resource and evaluation framework for studying how far ahead personal behavior can be anticipated from egocentric observation.

Problem

• personal behavior anticipation
• egocentric video
• action prediction
• open-vocabulary sequence prediction

Innovation

• Egocentric Video
• Action Anticipation
• Open-vocabulary Prediction
• Hierarchical Abstraction
• Soft Edit Distance