Model Multiplicity for Adversarial Detection in Small Language Model Training on Edge Devices

2026-06-05

AI Summary

This work addresses the vulnerability of distributed fine-tuning of small language models on edge devices to collaborative or persistent poisoning attacks, which conventional single-model defense mechanisms struggle to mitigate effectively. The authors propose a system-level defense framework that leverages model diversity by concurrently or sequentially training multiple small language models—such as DistilGPT-2—at the edge. By analyzing inter-model gradient similarity, loss evolution trajectories, and parameter variance, the approach identifies anomalous client behavior indicative of poisoning. This is the first method to exploit model diversity explicitly to enhance the sensitivity and robustness of poisoning detection. Empirical evaluations across heterogeneous settings and diverse attack scenarios demonstrate that the proposed mechanism achieves earlier and more reliable detection compared to state-of-the-art single-model defenses like Flanders and Robust.

Abstract

The rise of edge-based machine learning has enabled distributed adaptation of language models across mobile and IoT devices, offering privacy preservation and real-time responsiveness. However, distributed fine-tuning of language models on untrusted or heterogeneous edge nodes introduces new vulnerabilities. Compromised or unreliable devices can inject poisoned updates, leading to stealthy model manipulation or convergence degradation. Classical defenses such as robust aggregation or temporal anomaly detection operate on a single global model and are therefore limited in detecting coordinated or persistent poisoning. This work proposes a new system-level defense based on model multiplicity. Instead of maintaining one global model, the system rotates or concurrently trains multiple small language models (e.g., DistilGPT-2), each updated by independently sampled subsets of edge nodes. These models evolve under distinct training trajectories, creating multiple independent views of the same distributed population. Divergence between models quantified through gradient similarity, loss evolution, or parameter variance serves as a signal of anomalous or adversarial behavior. When one model deviates significantly from the ensemble mean, the system flags its contributing nodes for isolation or re-weighting. We implement this framework and evaluate it on edge-scale simulations of Small Language Model (SLM) training under varying heterogeneity and attack conditions. Results show that model multiplicity enables earlier and more reliable detection of poisoning compared to classical single-model defenses such as Flanders and Robust methods. Our findings demonstrate that diversity in model evolution can serve as a practical and effective defense mechanism for secure distributed learning on resource-constrained edge devices.

Problem

• model poisoning
• edge computing
• distributed learning
• adversarial detection
• small language models

Innovation

• model multiplicity
• adversarial detection
• edge-based learning
• distributed poisoning defense
• small language models