SERUM: State Extraction and Refinement for User Modeling

2026-07-31

AI Summary

Automatically constructing interpretable user behavior models from unstructured screen-recorded videos remains challenging. This work proposes SERUM, a framework that iteratively processes egocentric screen recordings by integrating a hierarchical vision-language model for activity recognition and intention inference, while leveraging contextual information to refine state labels. Synonymous states are aggregated using sentence embeddings and human-calibrated thresholds to yield a compact behavioral taxonomy. Notably, this is the first approach to generate structured process models without manual annotations, introducing the concept of “schema equilibrium” and proving that iterative refinement converges to a stable state vocabulary. Evaluation on 61 videos spanning four task categories—including programming and cooking—demonstrates that the normalized Markov model significantly outperforms frequency-based baselines, with human assessments confirming markedly improved label quality.

Abstract

Agentic assistants capable of proactive, personalized interactions require structured models of user intent and workflow. However, building these models from raw, unstructured screen activity remains an open challenge. We present SERUM, a multi-pass framework that extracts finite-state behavioral models directly from unstructured egocentric video using hierarchical VLM annotation. Processing screen recordings through a sliding window, SERUM alternates between activity-recognition and intent-inference passes, with each pass refining labels using accumulated prior context to reduce hallucination and temporal conflation seen in single-pass annotation. Synonymous states are then merged via sentence embeddings and human-calibrated thresholds into a compact, coherent taxonomy. We evaluate behavioral structure by fitting first-order Markov models over the resulting label sequences (both actions and intents) and measuring predictive accuracy against frequency baselines. Across 61 egocentric videos in four domains (coding, cooking, physical activities, and daily life), we find: (1) iterative label refinement converges to a stable state vocabulary, which we term schematic equilibrium, after several passes; (2) normalized Markov models achieve substantially lower perplexity and higher action predictions than frequency baselines, with the largest gains on structured tasks like coding; and (3) human annotators rate final-pass labels as accurate and meaningfully improved over first-pass labels. To our knowledge, SERUM is the first system to produce interpretable process models from unstructured egocentric screen video without manual annotation, opening a scalable pathway for user modeling and behavioral understanding in the wild. Our demo, code, and results are publicly available

Problem

• user modeling
• egocentric video
• behavioral modeling
• intent inference
• structured representation

Innovation

• state extraction
• user modeling
• egocentric video
• iterative refinement
• behavioral taxonomy