MCFuser: High-Performance and Rapid Fusion of Memory-Bound Compute-Intensive Operators

2025-06-27

AI Summary

Traditional fusion approaches for memory-constrained, compute-intensive dynamic tensor operator chains suffer from narrow search spaces, redundant memory accesses, and high tuning overhead. Method: This paper proposes MCFuser—a framework that (i) formally defines such operator chains; (ii) constructs a complete fusion strategy search space using high-dimensional tiling expressions; and (iii) integrates DAG-driven memory access optimization with an analytical performance model-guided heuristic search to enable efficient pruning and automatic kernel generation. Contribution/Results: Evaluated on NVIDIA A100 and RTX 3080 GPUs, MCFuser achieves up to 5.9× higher kernel performance and reduces tuning time by 70× compared to state-of-the-art compilers (e.g., Ansor). It significantly improves GPU data locality and alleviates memory bandwidth bottlenecks.

Technology Category

Machine Learning: Matrix & Tensor MethodsSearch and Optimization: Sampling/Simulation-based SearchConstraint Satisfaction and Optimization: Distributed CSP/Optimization

Application Category

User Modeling, Personalization and Recommendation: Federated recommendation systems and personalizationGraph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsSearch and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Operator fusion, a key technique to improve data locality and alleviate GPU memory bandwidth pressure, often fails to extend to the fusion of multiple compute-intensive operators due to saturated computation throughput. However, the dynamicity of tensor dimension sizes could potentially lead to these operators becoming memory-bound, necessitating the generation of fused kernels, a task hindered by limited search spaces for fusion strategies, redundant memory access, and prolonged tuning time, leading to sub-optimal performance and inefficient deployment. We introduce MCFuser, a pioneering framework designed to overcome these obstacles by generating high-performance fused kernels for what we define as memory-bound compute-intensive (MBCI) operator chains. Leveraging high-level tiling expressions to delineate a comprehensive search space, coupled with Directed Acyclic Graph (DAG) analysis to eliminate redundant memory accesses, MCFuser streamlines kernel optimization. By implementing guidelines to prune the search space and incorporating an analytical performance model with a heuristic search, MCFuser not only significantly accelerates the tuning process but also demonstrates superior performance. Benchmarked against leading compilers like Ansor on NVIDIA A100 and RTX3080 GPUs, MCFuser achieves up to a 5.9x speedup in kernel performance and outpaces other baselines while reducing tuning time by over 70-fold, showcasing its agility.

Problem

• Fusion of multiple compute-intensive operators hindered by computation saturation
• Dynamic tensor dimensions causing memory-bound operators requiring fused kernels
• Limited fusion strategy search spaces and redundant memory access degrading performance

Innovation

• High-level tiling expressions for search space
• DAG analysis to reduce memory access
• Analytical model with heuristic search