Remembering by Reconstructing: Domain Incremental Learning With Test-Time Training on Video Streams

2026-05-29

AI Summary

This work addresses the challenge of continual domain-incremental learning in non-stationary video streams by proposing a replay-free approach that jointly trains a primary task head with a self-supervised Masked Autoencoder (MAE). During incremental learning phases, the method learns domain-specific LoRA adapters for each new domain. At inference, it dynamically selects the optimal LoRA adapter through online test-time training of the MAE head, effectively enabling “reconstructive memory” of the current domain. The key innovation lies in actively leveraging—rather than suppressing—catastrophic forgetting by synergistically integrating LoRA, MAE, and test-time training. This strategy achieves significant performance gains on action recognition and semantic segmentation tasks, particularly in scenarios involving highly correlated and gradually drifting video streams.

Abstract

In this work we introduce a novel approach to domain incremental learning, adapting models over time to evolving, non-stationary data. In contrast to other works, we do not attempt to avoid catastrophic forgetting, but rather allow it and exploit it. Our model combines a main task head with a self-supervised masked autoencoder (MAE) head. We then learn domain-specific LoRA adapters during incremental training. Each adapter specializes to its domain, naturally inducing forgetting on other domains in both heads. At inference, we perform online test-time training on the self-supervised MAE head to identify which LoRAs best matches the current input, so the model can `remember' the domain again. Our scheme is especially well-suited to real-world streaming data, such as video, where consecutive samples are highly correlated and domain shifts are gradual. We demonstrate our method on domain-incremental action recognition and semantic segmentation tasks.

Problem

• domain incremental learning
• catastrophic forgetting
• video streams
• non-stationary data
• test-time training

Innovation

• domain incremental learning
• test-time training
• LoRA adapters
• masked autoencoder
• catastrophic forgetting