SoK: Colluding Adversaries in Machine Learning Pipelines

2026-06-08

AI Summary

This study addresses the lack of systematic analysis of collusion among diverse adversarial actors in machine learning, which hinders comprehensive assessment of their compounded risks to security, privacy, and fairness. The work proposes a unified framework that formally models collusion mechanisms both between training-time and inference-time attackers and among inference-time attackers themselves. Grounded in an identification of enabling factors, the framework yields principled guidelines for predicting collusion likelihood. Through a systematic literature review, theoretical analysis, and empirical validation across scenarios involving adversarial attacks, privacy breaches, and fairness violations, the study not only explains existing findings but also successfully hypothesizes and verifies five previously unexplored collusion patterns, revealing the critical influence of attacker characteristics on collusion potential.

Abstract

Machine learning (ML) models are susceptible to various security, privacy, and fairness risks. Adversaries with different characteristics (i.e., objectives, knowledge, and capabilities) can collude by executing one attack to amplify others. Existing work lacks a systematic framework to explore collusion among adversaries, and to study the implications of the adversaries' characteristics. We present a framework covering collusion (a) between train- and inference-time adversaries, and (b) among inference-time adversaries. Our framework accounts for factors enabling collusion between adversaries. We propose a guideline to conjecture about the potential for collusion using enabling factors. We use it to explain prior work, conjecture about unexplored collusions, and empirically validate five such cases. Finally, we discuss how adversaries' characteristics influence the potential for collusion.

Problem

• collusion
• adversaries
• machine learning pipelines
• security risks
• adversarial characteristics

Innovation

• adversarial collusion
• machine learning security
• inference-time attacks
• training-time attacks
• threat modeling