Joint Discriminative-Generative Modeling via Dual Adversarial Training

2025-10-13

AI Summary

Simultaneously achieving robust classification and high-fidelity generative modeling has long suffered from training instability and poor sample quality. To address this, we propose the first stable and efficient unified framework grounded in Energy-Based Models (EBMs), eliminating stochastic gradient Langevin dynamics (SGLD) sampling and instead optimizing the energy function via adversarial training. We introduce a contrastive sample discrimination mechanism based on binary cross-entropy (BCE) loss and a cooperative adversarial training strategy, which enhances classification robustness while obviating explicit gradient penalties. Furthermore, a two-stage training scheme mitigates incompatibility between batch normalization and EBMs. On CIFAR-10/100 and ImageNet, our method achieves state-of-the-art classification robustness among hybrid models, generates samples surpassing BigGAN and rivaling diffusion models in fidelity, and—crucially—demonstrates, for the first time, high-quality generation by MCMC-based EBMs on high-resolution, complex natural image datasets.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Large pretrained models with web data

Abstract

Simultaneously achieving robust classification and high-fidelity generative modeling within a single framework presents a significant challenge. Hybrid approaches, such as Joint Energy-Based Models (JEM), interpret classifiers as EBMs but are often limited by the instability and poor sample quality inherent in SGLD-based training. We address these limitations by proposing a novel training framework that integrates adversarial training (AT) principles for both discriminative robustness and stable generative learning. The proposed method introduces three key innovations: (1) the replacement of SGLD-based JEM learning with a stable, AT-based approach that optimizes the energy function by discriminating between real data and PGD-generated contrastive samples using the BCE loss; (2) synergistic adversarial training for the discriminative component that enhances classification robustness while eliminating the need for explicit gradient penalties; and (3) a two-stage training procedure to resolve the incompatibility between batch normalization and EBM training. Experiments on CIFAR-10, CIFAR-100, and ImageNet demonstrate that our method substantially improves adversarial robustness over existing hybrid models while maintaining competitive generative performance. On ImageNet, when optimized for generative modeling, our model's generative fidelity surpasses that of BigGAN and approaches diffusion models, representing the first MCMC-based EBM approach to achieve high-quality generation on complex, high-resolution datasets. Our approach addresses key stability issues that have limited JEM scaling and demonstrates that adversarial training can serve as an effective foundation for unified frameworks capable of generating and robustly classifying visual data.

Problem

• Achieving robust classification and generative modeling simultaneously
• Overcoming instability in Joint Energy-Based Models training
• Resolving incompatibility between batch normalization and EBM training

Innovation

• Replaces SGLD with adversarial training for stable learning
• Uses synergistic adversarial training to enhance robustness
• Implements two-stage training to resolve batch normalization issues