InfoOps Bench: A live information operations safety benchmark

2026-07-30

AI Summary

This study evaluates the safety and integrity of state-of-the-art large language models under state-sponsored information manipulation. Leveraging over 2,100 real-world disinformation cases attributed to official sources in Russia, China, and Iran, the authors construct the first dynamically updated benchmark platform, integrating weekly-tracked false claims to assess the response behaviors of 17 prominent models through four prompt frameworks. The work introduces a novel multi-dimensional prompt engineering approach and an automated integrity scoring system—measuring refusal rates, fact-checking frequency, and content harmfulness—to uncover complex trade-offs between compliance and factual accuracy. Results reveal an 85.7-percentage-point span in model integrity scores, independent of model scale; notably, certain Chinese-developed models exhibit a sharp 48–70 percentage point drop in compliance on China-critical content, with significant variations in generating fabricated details, downplaying claims, or proactively fact-checking.

Abstract

In this paper we present an active, constantly updated AI benchmark which measures the integrity of frontier language models against being co-opted for state-backed information operations. We draw on over 2,100 information operations from a live monitoring pipeline which tracks Russian, Chinese and Iranian state-backed information assets. Alongside this paper, we release a companion website that tracks the most prominent claims spread by state-backed media outlets, updated weekly, available from: pattrn.ai/research/infoopsbench. The dynamic nature of the benchmark makes it resistant to saturation. In the benchmark, we test 17 models from 8 providers across four prompt framings. We find that most models can be co-opted for information operations. Integrity scores, defined as the percentage of refused requests, range from 8.8% to 94.5%, an 85.7-percentage-point spread not explained by model size. Model choice also changes the character of the resulting operation. Some models fabricate details and produce output more harmful than the source material, others defuse claims even while complying, and fact-checking rates vary from 2.9% to 72.9%. Integrity against information operations is at least partly related to refusal to produce content even for benign claims, illustrating the challenge of balancing model usability with safety. With one exception (Z.ai's GLM 5.2), the Chinese-developed models sharply cut compliance on factually grounded but China-critical claims, dropping 48-70 percentage points relative to matched benign claims.

Problem

• information operations
• AI safety
• language models
• model integrity
• state-backed disinformation

Innovation

• information operations
• AI safety benchmark
• dynamic evaluation
• model integrity
• refusal behavior