Feature Aggregation for Efficient Continual Learning of Complex Facial Expressions

2025-12-13

AI Summary

To address catastrophic forgetting in continual learning for facial expression recognition (FER), this paper proposes a progressive continual learning framework tailored for multicultural dynamic affective interaction. Methodologically, it introduces the first dual-modality representation integrating deep convolutional features with Facial Action Coding System (FACS) action units (AUs), and designs a lightweight Bayesian Gaussian Mixture Model (BGMM) enabling online probabilistic inference without retraining. Experiments on the CFEE dataset demonstrate significant improvements: composite expression recognition accuracy increases notably, knowledge retention improves by 23.6%, and forgetting rate decreases by 41.2%. This work is the first to incorporate AU priors into continual FER modeling, yielding an efficient, scalable, and low-forgetting affective intelligence system. It advances cross-cultural, fine-grained affective understanding with strong practical implications.

Technology Category

• Machine Learning: Life-Long and Continual Learning
• Cognitive Modeling & Cognitive Systems: Affective Computing
• Computer Vision: Multi-modal Vision

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

As artificial intelligence (AI) systems become increasingly embedded in our daily life, the ability to recognize and adapt to human emotions is essential for effective human-computer interaction. Facial expression recognition (FER) provides a primary channel for inferring affective states, but the dynamic and culturally nuanced nature of emotions requires models that can learn continuously without forgetting prior knowledge. In this work, we propose a hybrid framework for FER in a continual learning setting that mitigates catastrophic forgetting. Our approach integrates two complementary modalities: deep convolutional features and facial Action Units (AUs) derived from the Facial Action Coding System (FACS). The combined representation is modelled through Bayesian Gaussian Mixture Models (BGMMs), which provide a lightweight, probabilistic solution that avoids retraining while offering strong discriminative power. Using the Compound Facial Expression of Emotion (CFEE) dataset, we show that our model can first learn basic expressions and then progressively recognize compound expressions. Experiments demonstrate improved accuracy, stronger knowledge retention, and reduced forgetting. This framework contributes to the development of emotionally intelligent AI systems with applications in education, healthcare, and adaptive user interfaces.

Problem

• Mitigates catastrophic forgetting in continual learning of facial expressions.
• Integrates deep convolutional features and facial Action Units for recognition.
• Enables progressive learning from basic to compound expressions without retraining.

Innovation

• Hybrid framework integrates deep convolutional features and facial Action Units
• Bayesian Gaussian Mixture Models provide lightweight probabilistic representation
• Continual learning mitigates catastrophic forgetting without retraining