Towards Better De-raining Generalization via Rainy Characteristics Memorization and Replay

2025-06-03

AI Summary

Existing image deraining methods suffer from limited generalization across diverse rain scenarios due to reliance on single-source training data. To address this, we propose a continual learning framework inspired by the Complementary Learning Systems theory in neuroscience: it employs a generative adversarial network (GAN) to model novel rain patterns—mimicking hippocampal memory—and integrates knowledge distillation with historical sample replay for parameter consolidation—emulating neocortical learning. The framework enables sequential training of deraining models across multiple heterogeneous rainy datasets without revisiting prior data. We validate it by continually training three state-of-the-art deraining architectures across six real-world rainy datasets, achieving substantial gains in cross-domain generalization and outperforming current SOTA methods. Our key contribution is the first incorporation of neurocognitive mechanisms into deraining continual learning, enabling dynamic rain-feature memory, selective parameter consolidation, and effective knowledge transfer across rain distributions.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

Current image de-raining methods primarily learn from a limited dataset, leading to inadequate performance in varied real-world rainy conditions. To tackle this, we introduce a new framework that enables networks to progressively expand their de-raining knowledge base by tapping into a growing pool of datasets, significantly boosting their adaptability. Drawing inspiration from the human brain's ability to continuously absorb and generalize from ongoing experiences, our approach borrow the mechanism of the complementary learning system. Specifically, we first deploy Generative Adversarial Networks (GANs) to capture and retain the unique features of new data, mirroring the hippocampus's role in learning and memory. Then, the de-raining network is trained with both existing and GAN-synthesized data, mimicking the process of hippocampal replay and interleaved learning. Furthermore, we employ knowledge distillation with the replayed data to replicate the synergy between the neocortex's activity patterns triggered by hippocampal replays and the pre-existing neocortical knowledge. This comprehensive framework empowers the de-raining network to amass knowledge from various datasets, continually enhancing its performance on previously unseen rainy scenes. Our testing on three benchmark de-raining networks confirms the framework's effectiveness. It not only facilitates continuous knowledge accumulation across six datasets but also surpasses state-of-the-art methods in generalizing to new real-world scenarios.

Problem

• Improving image de-raining generalization across diverse real-world conditions
• Expanding de-raining knowledge via continuous dataset integration and GANs
• Enhancing adaptability through hippocampal-inspired memory replay and distillation

Innovation

• GANs capture and retain new rainy features
• De-raining network trains with synthesized data
• Knowledge distillation enhances unseen scene performance