Mixture of Robust Experts (MoRE):A Robust Denoising Method towards multiple perturbations

2021-04-21

AI Summary

Deep neural networks exhibit insufficient robustness against diverse perturbations—including ℓ₁, ℓ₂, and ℓ∞ adversarial noise as well as natural corruptions (e.g., adverse weather)—and existing adversarial training methods suffer from limited generalization across perturbation types. Method: This paper proposes the Mixture of Robust Experts (MoRE) framework, the first to formulate multi-perturbation robust learning as a mixture-of-experts mechanism. MoRE decouples robustness objectives along distinct ℓₚ-norm perturbation directions for joint optimization, incorporates dynamic gating for expert selection, enables robust feature sharing, and employs joint task training. Contribution/Results: Evaluated on CIFAR-10/100 and an ImageNet subset, MoRE significantly improves robust accuracy under mixed ℓₚ perturbations—achieving an average gain of +6.2% over unified adversarial training—while preserving clean-input accuracy. It overcomes the inflexibility of single-norm adversarial paradigms, enabling adaptive, cross-norm robustness without compromising standard performance.

Abstract

To tackle the susceptibility of deep neural networks to examples, the adversarial training has been proposed which provides a notion of robust through an inner maximization problem presenting the first-order embedded within the outer minimization of the training loss. To generalize the adversarial robustness over different perturbation types, the adversarial training method has been augmented with the improved inner maximization presenting a union of multiple perturbations e.g., various $ell_p$ norm-bounded perturbations.

Problem

• Enhancing robustness against multiple adversarial perturbations
• Dynamic expert weighting for diverse data types
• Addressing obfuscated gradients via joint gating training

Innovation

• Gating mechanism assembles diverse expert networks
• Dynamic weight assignment for various data types
• Adversarial training fine-tunes gating and experts