memory-efficient implementation

This work addresses the GPU memory bottleneck in Transformer models caused by high parameter and activation memory demands during training and inference. The authors propose a novel parallelism strategy that integrates tensor parallelism (TP) and sequence parallelism (SP) along the same device axis, enabling each device to simultaneously shard both model weights and input sequences. By leveraging broadcast-based weight sharding with key-value exchange in attention layers and ring-based weight passing with local accumulation in gated MLPs, the method achieves dual compression of both parameter and activation memory. This approach significantly reduces per-device memory consumption, outperforming conventional TP, SP, and their hybrid variants. It demonstrates superior hardware adaptability and scaling efficiency under long-context and memory-constrained settings, while seamlessly integrating with pipeline and expert parallelism.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

To address the high energy consumption caused by data movement in large language model (LLM) inference, this paper proposes an efficient block-wise sparsification method tailored for Transformer linear layers. We introduce a novel, hardware-agnostic block sparsity pattern enabling up to 95% sparsity with negligible accuracy degradation; further, we integrate iterative sparsification with a highly optimized fused sparse matrix-matrix multiplication (SpMM) kernel. Our approach delivers end-to-end acceleration across diverse hardware architectures and datasets. Experiments show up to 16.7× speedup for MLP computation, 1.6× end-to-end inference acceleration, 1.11× pretraining speedup, and a 3.12× reduction in inference memory footprint. Crucially, the method achieves a superior trade-off among accuracy, throughput, and energy efficiency—establishing a new paradigm for efficient LLM deployment.

To address efficient deployment of large language and vision models on resource-constrained devices, this work systematically investigates the non-orthogonality between sparsification and quantization—two key model compression techniques—and characterizes their joint accuracy degradation mechanism. We provide the first rigorous mathematical proof of their non-orthogonality, revealing that error accumulation is intrinsic and critically dependent on operational ordering: quantizing before sparsifying severely degrades accuracy, whereas sparsifying before quantizing mitigates error propagation. Grounded in theoretical analysis and validated across diverse models (OPT, LLaMA-125M–8B, ViT, ResNet) and hardware platforms, we establish “sparsify-then-quantize” as the optimal practice. This principle achieves high compression ratios while markedly improving the accuracy–efficiency trade-off, offering both theoretically grounded insights and a practical, empirically verified paradigm for edge deployment of foundation models.

This work addresses efficient decomposition of high-dimensional sparse tensors—common in healthcare and cybersecurity—on modern parallel processors, overcoming restrictive assumptions about mode structure or sparsity distribution inherent in conventional compressed formats. We propose ALTO, an adaptive linearization tensor representation that is agnostic to both mode structure and sparsity distribution. Built upon ALTO, we design a parallel decomposition algorithm featuring low synchronization overhead and high data reuse, augmented by dynamic performance modeling and scheduling heuristics for automatic hardware adaptation. Leveraging cache- and memory-aware optimizations on Intel Xeon Scalable platforms, experiments demonstrate that ALTO achieves over 10× speedup versus the best structure-agnostic format and a 5.1× geometric mean speedup versus the best structure-aware format, while incurring only 25% of the latter’s storage overhead.

This study addresses the lack of systematic evaluation of tensor decomposition methods for post-training compression of large language models (LLMs), particularly regarding their applicability to both dense and mixture-of-experts (MoE) architectures. Through a combination of theoretical analysis and empirical experiments, this work provides the first comprehensive assessment of tensor decomposition across diverse LLM architectures, revealing critical performance trade-offs. It identifies a fundamental mismatch between the shared subspace assumption inherent in tensor decomposition and the heterogeneous representations actually learned by modern LLMs. By delineating the practical boundaries and limitations of tensor-based compression in contemporary LLMs, this research offers actionable insights for efficient model deployment and releases its implementation code to facilitate further investigation.

This work addresses the performance limitations of data-intensive applications in edge computing, which often suffer from poor memory locality or excessive memory footprint. To overcome these challenges, the authors propose a hardware-software co-design that integrates a tensor memory engine into the general-purpose CPU datapath. This engine enables runtime dynamic reconfiguration of memory layouts to significantly enhance cache locality, without requiring application code modifications, offloading computation to memory, or incurring memory overhead. By preserving the CPU’s role as the primary compute unit, the approach effectively improves overall system performance. Notably, this study presents the first practical implementation of runtime memory layout reconfiguration on commercial SoC/FPGA platforms, demonstrating both innovation and real-world applicability.

This work proposes a novel approach to efficiently execute sparse neural networks on dense matrix-multiplication accelerators, eschewing conventional specialized sparse architectures. By optimizing the layout and scheduling of pruned sparse data, the method enables the deployment of a greater number of dense processing elements (PEs) within the same hardware area, thereby significantly improving hardware utilization. The elimination of complex index-matching circuitry—typically required in dedicated sparse accelerators—reduces both area and power overheads. Experimental results demonstrate that the proposed technique outperforms specialized sparse accelerators in both area efficiency and energy efficiency, achieving dual optimization of computational performance.

This work addresses the challenge of deploying mainstream deep learning frameworks in resource-constrained environments, where their large size and lack of lightweight yet fully featured alternatives pose significant limitations. To this end, we propose and implement a lightweight tensor computation library built in Rust, leveraging its performance and memory safety guarantees to construct an efficient computational engine. The system exposes a PyTorch-like Python interface via PyO3, supporting essential features including n-dimensional tensors, dynamic computation graphs, reverse-mode automatic differentiation, neural network layers, and optimizers. The resulting installable package occupies only a few megabytes—orders of magnitude smaller than PyTorch or TensorFlow—while retaining the core capabilities necessary for research and development on CPU-based systems.

This work addresses the limitation of Transformers in lacking an explicit knowledge storage mechanism, which hinders efficient retention and retrieval of learned information. To overcome this, the authors propose a chaptered sparse memory bank, where learnable memory tokens are queried by the Transformer via cross-attention. Inspired by Mixture-of-Experts, a dynamic chapter routing strategy selectively activates relevant subsets of memory, enabling scalable knowledge access while maintaining computational efficiency. The approach expands memory capacity to 262K tokens—introducing a new scaling dimension beyond model parameters—without incurring prohibitive computational overhead. Experiments demonstrate that, under matched FLOPs, the proposed model outperforms standard Transformers in both pretraining and instruction fine-tuning tasks, while also exhibiting substantially improved knowledge retention and robustness against catastrophic forgetting in continual learning scenarios.