CBIL: Collective Behavior Imitation Learning for Fish from Real Videos

2024-11-19

ACM Transactions on Graphics

AI Summary

This work addresses the challenge of modeling collective behaviors in high-density, irregular fish schools. We propose the first end-to-end, video-driven self-supervised imitation learning framework that learns spatiotemporal motion patterns directly from raw videos—without requiring ground-truth trajectory annotations. Methodologically, the framework integrates a Masked Video Autoencoder (MVAE) with self-supervised video representation learning to extract robust spatiotemporal features; introduces an adversarial latent-space motion distribution matching mechanism; and incorporates biologically inspired reward functions and prior motion constraints to enhance training stability and behavioral plausibility. Experiments demonstrate substantial improvements in behavioral diversity and visual fidelity of generated motions. The framework generalizes effectively to multi-species animated synthesis and enables automatic detection of anomalous schooling behaviors in field-captured videos.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Computer Vision: Video Understanding & Activity Analysis
• Humans and AI: Human-Aware Planning and Behavior Prediction

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Search and Retrieval-Augmented AI: Retrieval-Augmented Generation (RAG) and multi-modal RAG
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Reproducing realistic collective behaviors presents a captivating yet formidable challenge. Traditional rule-based methods rely on hand-crafted principles, limiting motion diversity and realism in generated collective behaviors. Recent imitation learning methods learn from data but often require ground-truth motion trajectories and struggle with authenticity, especially in high-density groups with erratic movements. In this paper, we present a scalable approach, Collective Behavior Imitation Learning (CBIL), for learning fish schooling behavior directly from videos , without relying on captured motion trajectories. Our method first leverages Video Representation Learning, in which a Masked Video AutoEncoder (MVAE) extracts implicit states from video inputs in a self-supervised manner. The MVAE effectively maps 2D observations to implicit states that are compact and expressive for following the imitation learning stage. Then, we propose a novel adversarial imitation learning method to effectively capture complex movements of the schools of fish, enabling efficient imitation of the distribution of motion patterns measured in the latent space. It also incorporates bio-inspired rewards alongside priors to regularize and stabilize training. Once trained, CBIL can be used for various animation tasks with the learned collective motion priors. We further show its effectiveness across different species. Finally, we demonstrate the application of our system in detecting abnormal fish behavior from in-the-wild videos.

Problem

• Learn fish schooling behavior from videos without motion trajectories
• Extract implicit states from videos using Masked Video AutoEncoder
• Detect abnormal fish behavior in real-world videos

Innovation

• Uses Masked Video AutoEncoder for self-supervised learning
• Adversarial imitation learning captures complex fish movements
• Incorporates bio-inspired rewards to stabilize training