Agent-Centric Animal Pose Forecasting

2026-07-21

AI Summary

This work proposes an agent-centric autoregressive generative model to algorithmically elucidate the mechanisms of perception, internal modeling, planning, and action generation in individual and collective animal behavior. Operating in an egocentric coordinate frame, the model processes pose trajectory data and predicts discretized action sequences to naturally emulate the behavioral constraints inherent to an organism’s self-referential perspective. The framework supports parallel multi-agent representations, enabling complex social behaviors to emerge spontaneously from each agent’s independent perception of and response to others. An accompanying open-source library provides composable action representations and systematic evaluation tools. Experiments demonstrate that the model accurately captures the distribution of social behaviors in courting fruit fly groups and exhibits strong cross-domain transferability.

Abstract

Understanding animal behavior at an algorithmic level -- what animals attend to, how they form internal models and plans, and how this maps to action -- remains a central challenge in neuroscience and ethology. Data-driven generative models offer a path toward this understanding. We introduce a framework for training agent-centric autoregressive models of animal behavior from tracked pose, applicable to single animals and to groups in which each agent senses and responds to its conspecifics. Our models input egocentric sensory observations and output egocentric movements, mirroring the biological constraint that animals observe and act on the world from their own reference frame. Social behavior emerges from agents independently sensing and responding to one another. This agent-centric formulation requires managing many parallel representations of the same data, along with ML-specific transformations like discretization. We release a general-purpose library focused on the composable sequences of operations that translate between these representations. We show that trained models capture the distribution of social behavior in groups of courting Drosophila, and our library includes quantitative tools for measuring fit. We demonstrate how the library supports systematic comparison across input and output representations and that it adapts straightforwardly to a new domain.

Problem

• animal behavior
• agent-centric modeling
• pose forecasting
• social interaction
• egocentric representation

Innovation

• agent-centric modeling
• egocentric representation
• autoregressive behavior forecasting
• social interaction emergence
• composable transformation library