Towards Decentralized and Sustainable Foundation Model Training with the Edge

2025-07-02

AI Summary

Foundation model training faces dual challenges of high carbon emissions and centralized monopolization. This paper proposes a decentralized, green training paradigm leveraging collaborative edge devices—harnessing abundant idle computational resources from end-user terminals—by integrating federated learning, distributed optimization, and energy-aware scheduling. The method enables environmentally sustainable foundation model training with reduced ecological impact. Key contributions include: (1) the first systematic feasibility and sustainability analysis of large-model training via edge collaboration; (2) a holistic training architecture jointly optimizing communication efficiency, energy constraints, and statistical heterogeneity; and (3) identification of critical research challenges—including edge heterogeneity modeling, lightweight gradient aggregation, and cross-device privacy-utility trade-offs. Experimental results demonstrate substantial reduction in training carbon footprint while mitigating dependence on centralized compute infrastructure and data silos.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Humans and AI: Other Foundations of Human Computation & AI
• Computer Vision: Other Foundations of Computer Vision

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Sustainability and carbon-aware systems for Web, mobile, and WoT
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

Foundation models are at the forefront of AI research, appealing for their ability to learn from vast datasets and cater to diverse tasks. Yet, their significant computational demands raise issues of environmental impact and the risk of centralized control in their development. We put forward a vision towards decentralized and sustainable foundation model training that leverages the collective compute of sparingly used connected edge AI devices. We present the rationale behind our vision, particularly in support of its sustainability benefit. We further outline a set of challenges that need to be addressed to turn this vision into reality.

Problem

• Decentralized training of foundation models using edge devices
• Reducing environmental impact of large-scale AI model training
• Addressing centralized control risks in foundation model development

Innovation

• Decentralized training using edge AI devices
• Leveraging spare compute for sustainability
• Reducing centralized control in AI development