Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions

2025-03-20

AI Summary

This paper addresses core challenges in scalability, privacy preservation, and edge-resource coordination for distributed large language models (LLMs) and multimodal LLMs (MLLMs). It systematically surveys decentralized techniques across training, inference, fine-tuning, and deployment. Methodologically, it introduces the first comprehensive taxonomy for distributed MLLMs—spanning six dimensions: data/model/pipeline parallelism, federated learning, edge-coordinated inference, multimodal alignment compression, and privacy-enhancing computation (e.g., differential privacy, secure aggregation). The analysis identifies 12 critical bottlenecks, including weak robustness, insufficient privacy guarantees, and lack of cross-modal edge coordination. Key contributions include: (1) the first holistic architectural map of distributed MLLM technologies; (2) seven actionable research directions; and (3) a novel paradigm enabling scalable, secure, and cross-modal-fused distributed AI—providing both theoretical foundations and practical guidelines for industrial-grade deployment.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Language models (LMs) are machine learning models designed to predict linguistic patterns by estimating the probability of word sequences based on large-scale datasets, such as text. LMs have a wide range of applications in natural language processing (NLP) tasks, including autocomplete and machine translation. Although larger datasets typically enhance LM performance, scalability remains a challenge due to constraints in computational power and resources. Distributed computing strategies offer essential solutions for improving scalability and managing the growing computational demand. Further, the use of sensitive datasets in training and deployment raises significant privacy concerns. Recent research has focused on developing decentralized techniques to enable distributed training and inference while utilizing diverse computational resources and enabling edge AI. This paper presents a survey on distributed solutions for various LMs, including large language models (LLMs), vision language models (VLMs), multimodal LLMs (MLLMs), and small language models (SLMs). While LLMs focus on processing and generating text, MLLMs are designed to handle multiple modalities of data (e.g., text, images, and audio) and to integrate them for broader applications. To this end, this paper reviews key advancements across the MLLM pipeline, including distributed training, inference, fine-tuning, and deployment, while also identifying the contributions, limitations, and future areas of improvement. Further, it categorizes the literature based on six primary focus areas of decentralization. Our analysis describes gaps in current methodologies for enabling distributed solutions for LMs and outline future research directions, emphasizing the need for novel solutions to enhance the robustness and applicability of distributed LMs.

Problem

• Address scalability challenges in distributed large language models
• Explore privacy concerns with sensitive data in LM training
• Survey decentralized techniques for multimodal LM training and inference

Innovation

• Distributed computing for scalable language models
• Decentralized techniques for privacy and edge AI
• Multimodal data integration in distributed LLMs