Improving the Robustness/Accuracy Tradeoff Against Adversarial Attacks Using Information Bottleneck Distillation Through Dual Teachers

2026-07-30

AI Summary

This work proposes a dual-teacher information bottleneck distillation framework to mitigate the trade-off between adversarial robustness and clean accuracy. The method uniquely combines a teacher model trained solely on clean data with a robust teacher obtained through adversarial training, and introduces a cross-layer attention mechanism to fuse their features for guiding student learning. Experiments on CIFAR-10 and CIFAR-100 demonstrate that the proposed approach significantly improves clean accuracy while maintaining strong adversarial robustness, achieving state-of-the-art performance in terms of the harmonic mean of clean and robust accuracy.

Abstract

Deep neural networks (DNNs) have achieved remarkable success in classical machine learning problems. However, they are known to be vulnerable to adversarial attacks. Countermeasures proposed in the literature, notably Information Bottleneck Distillation (IBD) introduced by Kuang et al., degrade the classification accuracy on clean inputs while improving the robustness to adversarial inputs. In this work, we extend the IBD framework by introducing an extra teacher model (clean teacher) trained with only clean inputs, into the distillation process from a robust teacher model trained by adversarial training. The features of both clean and robust teachers are transferred to the student through a cross-layer attention matrix. Experimental results on the CIFAR-10 and CIFAR-100 datasets show that the proposed method improves classification accuracy on clean samples compared to the original IBD, while maintaining similar accuracy on adversarial samples. Furthermore, our methods are competitive with state-of-the-art approaches, including the recent dual-teacher distillation framework B-MTARD, particularly in terms of the harmonic mean between clean and robust accuracy. We also analyze the impact of different training settings that have different influences on the attention module.

Problem

• adversarial attacks
• robustness-accuracy tradeoff
• information bottleneck distillation
• dual teachers
• deep neural networks

Innovation

• Information Bottleneck Distillation
• Dual Teacher
• Adversarial Robustness
• Cross-layer Attention
• Robustness-Accuracy Tradeoff