Learning to Attack and Defend: Adaptive Red Teaming of Language Models via GRPO

2026-06-08

AI Summary

This work addresses the challenges of sustaining red-teaming evaluations against the dynamic evolution of language model attack and defense strategies, as well as the instability of GRPO in collaborative training settings. The authors propose AdvGRPO, a novel framework that enables the first stable application of GRPO in attacker–defender joint optimization. Stability is achieved through dense multi-channel reward modeling and decoupled normalization of advantage functions. Additionally, the framework incorporates a progressive curriculum learning strategy that advances from single-turn to closed-loop multi-turn attacks, integrated within an alternating co-training mechanism. Experimental results demonstrate that the generated attack strategies are both highly effective and transferable, and that the co-trained defensive models significantly outperform existing baselines on standard safety benchmarks.

Abstract

AI red teaming must continually adapt to evolving attackers and defenders. Reinforcement learning offers a promising approach to discovering novel attacks, and co-training methods can produce more robust defenders in tandem. Recent works have demonstrated the efficacy of attacker-defender co-training by applying PPO and DPO, but report that GRPO is unstable in this setting. We introduce AdvGRPO, a co-training framework that makes GRPO viable for joint attacker-defender optimization using dense multi-channel rewards and decoupled advantage normalization. Training progresses through a curriculum from single-turn to closed-loop multi-turn attacks before bootstrapping co-training, where attacker and defender models are updated in alternation. We show that our method can produce highly effective and transferable attacks and that co-trained defenders outperform baselines on safety benchmarks.

Problem

• AI red teaming
• language models
• adversarial attacks
• defender robustness
• co-training

Innovation

• AdvGRPO
• co-training
• red teaming
• multi-turn attacks
• advantage normalization