🤖 AI Summary
Deploying the Qwen2.5-0.5B model on the Xilinx Kria KV260 edge platform (ARM Cortex-A53 + FPGA) faces critical bottlenecks in computational demand, memory bandwidth, and power consumption. To address these challenges, this work proposes a co-optimization framework integrating Activation-aware Weight Quantization (AWQ) with a custom FPGA pipeline, alongside a CPU–FPGA hybrid execution paradigm and dynamic offloading scheduling strategy. The approach jointly optimizes quantization and hardware acceleration to preserve model accuracy while significantly reducing resource requirements. Experimental results demonstrate a 55.08% model compression rate and an inference throughput of 5.1 tokens/s—82% higher than the baseline. This yields substantial improvements in deployment efficiency and energy efficiency for large language models on resource-constrained edge devices.
📝 Abstract
Transformer-based Large Language Models (LLMs) have significantly advanced AI capabilities but pose considerable challenges for deployment on edge devices due to high computational demands, memory bandwidth constraints, and energy consumption. This paper addresses these challenges by presenting an efficient framework for deploying the Qwen2.5-0.5B model on the Xilinx Kria KV260 edge platform, a heterogeneous system integrating an ARM Cortex-A53 CPU with reconfigurable FPGA logic. Leveraging Activation-aware Weight Quantization (AWQ) with FPGA-accelerated execution pipelines, the proposed approach enhances both model compression rate and system throughput. Additionally, we propose a hybrid execution strategy that intelligently offloads compute-intensive operations to the FPGA while utilizing the CPU for lighter tasks, effectively balancing the computational workload and maximizing overall performance. Our framework achieves a model compression rate of 55.08% compared to the original model and produces output at a rate of 5.1 tokens per second, outperforming the baseline performance of 2.8 tokens per second.