data partitioning

This work addresses the inefficiency of conventional knowledge distillation methods in high-performance computing (HPC) systems, which employ symmetric parallel strategies for teacher and student models despite their markedly different memory and communication demands. To overcome this limitation, we propose an HPC-oriented asymmetric knowledge distillation framework that introduces, for the first time, a topology-aware heterogeneous parallelism mechanism. By combining vertical and horizontal model partitioning, our approach decouples the parallelization strategies of the teacher and student models and derives an analytical expression for the optimal partition-switching inflection point. Integrated with communication optimization and memory-reduction techniques, the proposed method achieves up to a 67% throughput improvement on real-world HPC clusters, substantially accelerating large-scale distillation training.

This work proposes a reverse curriculum learning framework tailored for structurally complex and conceptually deep tasks such as advanced mathematical problem solving and code generation. The approach introduces a difficulty scoring mechanism based on structural complexity and conceptual depth, and employs a teacher–student architecture to recursively decompose challenging problems. The teacher model generates progressively simplified examples through step-by-step reasoning, thereby constructing an easy-to-hard curriculum that guides the student model in incremental learning. Experimental results on benchmarks including MATH and AIME demonstrate that this method significantly outperforms standard training strategies, effectively enhancing the model’s capacity to solve complex problems.

To address the lack of unified benchmarks for model performance evaluation on high-dimensional big data in both local and distributed environments, this work designs an end-to-end evaluation framework covering three representative tasks—Epsilon (numerical regression), RestMex (text classification), and IMDb (movie feature analysis). Leveraging Apache Spark (Scala), we establish a reproducible heterogeneous computing experimental infrastructure to systematically compare traditional machine learning and deep learning models across accuracy, training efficiency, and resource consumption. This study presents the first pedagogically implemented standardized benchmark supporting multiple models, multimodal data, and diverse deployment scenarios, empirically uncovering performance bottlenecks and architectural trade-offs inherent in distributed scaling. The outcomes include an open-source evaluation pipeline, a standardized reporting template, and a reusable teaching paradigm—providing empirical foundations for AI system selection and optimization in big data contexts.

To address communication redundancy and computational load imbalance caused by sequence-length heterogeneity in long-context LLM training, this paper proposes Hybrid Data Parallelism (HDP), a framework that jointly optimizes communication and computation via dynamic grid-based communication, data-aware sequence sharding, selective gradient/activation offloading, and parallelism-aware load-balancing scheduling. HDP enables adaptive coordination between communication and computation across heterogeneous sequence lengths. To our knowledge, it is the first system to support efficient joint training of mixed-length sequences on ultra-large-scale clusters exceeding 12,000 GPUs. Evaluated on models ranging from 7B to 141B parameters and context lengths from 256K to 2048K tokens, HDP achieves up to 7.89× higher throughput compared to state-of-the-art systems, demonstrating substantial improvements in scalability and efficiency for long-context LLM training.

To address computational load imbalance in large Transformer model training—caused by skewed sequence-length distributions and the mismatch between linear memory complexity and quadratic computation complexity of attention mechanisms—this paper proposes a dynamic heterogeneous parallel training paradigm. Our method features a two-stage sequence-aware data allocation mechanism that jointly optimizes intra- and inter-group load balancing, integrated with cross-iteration workload modeling, dynamic scheduling, and heterogeneous GPU-aware resource allocation. To the best of our knowledge, this is the first approach to achieve joint optimization of computation, memory, and hardware resources for variable-length sequence training. Evaluated on real-world large-model training workloads, it delivers 1.32–2.66× speedup over baseline methods, significantly improving GPU utilization and training throughput.

This work addresses the longstanding limitation of large language model (LLM) training, which has been confined to high-performance data centers and unable to leverage the vast pool of heterogeneous, unreliable consumer-grade GPUs available across the internet. To overcome this, the authors introduce Agora, a novel system that pioneers a “protocol learning” paradigm, enabling permissionless, decentralized collective pretraining through communication-efficient pipeline parallelism, asynchronous optimization, and robust fault tolerance. In this framework, no participant holds the full model; instead, each retains only a shard of the parameters, ensuring collective ownership and inherent openness. The system successfully trained Pluralis-8B (8.6 billion parameters) on 500 billion FineWeb-Edu tokens using 330 dynamically joining and leaving consumer GPUs over 40 days, achieving 63% of the computational efficiency of a centralized baseline while matching its convergence performance closely.

Existing context parallelism approaches suffer from load imbalance, redundant communication, and suboptimal computational efficiency. This work proposes a communication-efficient and load-balanced context parallel training framework featuring three key innovations: a Whole-Doc document-level sharding strategy integrated with a Per-Doc hybrid sharding mechanism, a shard-aware communication protocol that effectively eliminates redundant transmission of KV caches, and a heuristic search-based near-optimal sharding planning algorithm. Experimental results demonstrate that the proposed method achieves up to a 1.63× speedup over state-of-the-art context parallelism techniques across multiple datasets.

This work addresses the GPU memory bottleneck in Transformer models caused by high parameter and activation memory demands during training and inference. The authors propose a novel parallelism strategy that integrates tensor parallelism (TP) and sequence parallelism (SP) along the same device axis, enabling each device to simultaneously shard both model weights and input sequences. By leveraging broadcast-based weight sharding with key-value exchange in attention layers and ring-based weight passing with local accumulation in gated MLPs, the method achieves dual compression of both parameter and activation memory. This approach significantly reduces per-device memory consumption, outperforming conventional TP, SP, and their hybrid variants. It demonstrates superior hardware adaptability and scaling efficiency under long-context and memory-constrained settings, while seamlessly integrating with pipeline and expert parallelism.

Existing large-scale model training systems struggle to flexibly compose diverse parallelization strategies, often relying on manual expert tuning and lacking generality. This work proposes a programmable distributed training system that enables users to declaratively specify composite parallelism strategies—such as data, pipeline, and expert parallelism—through model annotations and scheduling directives. These specifications are compiled via a unified intermediate representation (IR) into device-level execution plans, fully decoupling strategy definition from runtime execution over a global compute-communication DAG. The system is the first to support automatic compilation of user-defined composite strategies, matching the performance of established approaches like ZeRO while significantly improving both performance and memory efficiency in complex scenarios such as DeepSeek-V3’s DualPipe.

This work addresses the challenges of poor scalability and accuracy degradation in scientific machine learning when handling extremely high-resolution data, particularly due to the absence of a general-purpose parallelization framework supporting sub-unit batch sizes per device. The authors propose ShardTensor, a novel domain-parallel paradigm that shards tensors along spatial domains, thereby decoupling data dimensions from hardware constraints. This approach enables, for the first time, general-purpose parallel training and inference with sub-unit batch sizes. By supporting multidimensional parallelism and integrating dynamic computation–communication load balancing, ShardTensor simultaneously achieves strong scaling—reducing latency—and weak scaling—enabling larger-scale data processing—thereby significantly enhancing the scalability and efficiency of high-fidelity scientific computing tasks.