Scaling Decentralized Learning with FLock

2025-07-21

AI Summary

To address single-point failures, high communication overhead, and backdoor poisoning attacks in fine-tuning 70B-parameter large language models (LLMs) within decentralized environments, this paper proposes FLock: the first fully decentralized, collaborative fine-tuning framework integrating federated learning, blockchain consensus, verifiable computation, and on-chain economic incentives. FLock eliminates the central server, enabling secure and verifiable model update aggregation in trustless, multi-domain, heterogeneous settings—thereby removing all single-point attack surfaces. Experiments demonstrate a >68% improvement in defense success rate against backdoor poisoning attacks and significantly superior cross-domain generalization of the global model compared to isolated training baselines. This work achieves, for the first time, robust, third-party-free collaborative fine-tuning of a 70B-LLM, establishing a novel paradigm for open, cooperative AI training.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Coordination and Collaboration
• Natural Language Processing: Safety and Robustness

Application Category

• Economics, Online Markets and Human Computation: Architectures and workflows that use LLMs for crowd work
• Security and Privacy: Blockchains and distributed ledgers
• Systems and Infrastructure for Web, Mobile and WoT: Decentralized Web and Fediverse systems

Abstract

Fine-tuning the large language models (LLMs) are prevented by the deficiency of centralized control and the massive computing and communication overhead on the decentralized schemes. While the typical standard federated learning (FL) supports data privacy, the central server requirement creates a single point of attack and vulnerability to poisoning attacks. Generalizing the result in this direction to 70B-parameter models in the heterogeneous, trustless environments has turned out to be a huge, yet unbroken bottleneck. This paper introduces FLock, a decentralized framework for secure and efficient collaborative LLM fine-tuning. Integrating a blockchain-based trust layer with economic incentives, FLock replaces the central aggregator with a secure, auditable protocol for cooperation among untrusted parties. We present the first empirical validation of fine-tuning a 70B LLM in a secure, multi-domain, decentralized setting. Our experiments show the FLock framework defends against backdoor poisoning attacks that compromise standard FL optimizers and fosters synergistic knowledge transfer. The resulting models show a >68% reduction in adversarial attack success rates. The global model also demonstrates superior cross-domain generalization, outperforming models trained in isolation on their own specialized data.

Problem

• Decentralized LLM fine-tuning lacks control and efficiency
• Standard FL has central server vulnerability to attacks
• Scaling secure 70B-parameter LLM fine-tuning remains unsolved

Innovation

• Decentralized framework for LLM fine-tuning
• Blockchain-based trust with economic incentives
• Secure multi-domain decentralized validation