Hardware-Aware Data and Instruction Mapping for AI Tasks: Balancing Parallelism, I/O and Memory Tradeoffs

2025-09-03

AI Summary

To address the challenge of jointly optimizing I/O, memory, and parallelism in deep learning inference, this paper proposes a hardware-aware co-mapping framework for data and instructions. The method unifies the scheduling of data and instruction flows, integrating fine-grained message passing, weight residency and reuse, intra-array multicast, and staged reduction—enabling streaming on-chip execution on a message-based programmable architecture. Experimental results demonstrate that, on VGG-19, the design achieves 88–92% computational utilization, generates over 97% of messages on-chip, performs 89% of data transfers on-chip, reduces per-layer data movement by up to 100 MB, and delivers computation throughput exceeding 1 TFLOP/s. These improvements significantly alleviate reliance on host control, off-chip I/O, and DRAM access.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

We introduce a mapping framework for deep learning inference that takes advantage of predictable neural network behavior to plan both computation and communication ahead of time. The framework generates a unified stream of instructions and data, enabling the hardware to execute operations and route information on its own, without frequent involvement from the host and with minimal off-chip memory use. This naturally reduces reliance on I/O, off-chip memory, and host control. By leveraging fine-grained message passing on a programmable, message-based compute architecture, the framework keeps data movement local and coordinates computation across the array using techniques such as stationary-weight reuse, in-array multicasting, and staged reductions. Applied to VGG-19, the framework sustains high utilization (88 to 92 percent), with over 97 percent of messages generated internally and nearly 89 percent of time consumed on-chip transfers. Computation throughput scales beyond 1 TFLOP/s on larger arrays, while traffic reductions from reuse and local aggregation reach up to 100 MB per layer. Overall, the results highlight the effectiveness of streaming-based computation and show how our mapper enables this execution style by tightly coordinating data and instruction flow across the hardware.

Problem

• Optimizing hardware-aware mapping for AI inference tasks
• Balancing parallelism, I/O, and memory tradeoffs in deep learning
• Reducing off-chip memory use and host control dependency

Innovation

• Pre-planned computation and communication for neural networks
• Unified instruction and data stream minimizing host involvement
• Fine-grained message passing enabling local data movement