Sparse by Command: Task-Conditional Compute Skipping for Multi-Task Inference Accelerators

2026-07-24

AI Summary

This work addresses the inefficiency and latency in multitask reasoning models caused by executing computations irrelevant to the current task. The authors propose a task-instruction-driven conditional sparsity mechanism that employs a lightweight gating network to predict task-specific block-wise binary masks, dynamically skipping unnecessary computations without altering the backbone model architecture. A co-designed hardware-software system supports this approach, featuring an instruction set enabling mask-based skipping, a configurable block-wise INT8 accelerator, and a double-buffered memory system. Evaluated on the CARLA autonomous driving simulator, the method reduces computational load by 66–76%, achieves end-to-end inference latency of 3.74–4.44 ms (a 2.1–2.4× speedup), and lowers per-inference energy consumption from 263 mJ to 108–128 mJ.

Abstract

Multi-task inference models share a single backbone across diverse tasks, yet execute identical computation regardless of which task is active - wasting energy and cycles on task-irrelevant operations. We observe that the task command, typically available before inference begins, provides a free signal that can be exploited to skip unnecessary computation at the hardware level. We present a HW/SW co-designed approach in which a lightweight gating network, trained jointly with the backbone, predicts per-tile binary execution masks conditioned on the task input. Each tile corresponds to a fixed group of output channels (the native scheduling granularity of the accelerator), enabling masked tiles to be skipped with zero overhead. This yields a task-dependent reduction in compute, where each command activates only the subset of the network it requires, without changes to the model architecture or inference pipeline. We co-design the full system stack: a command-conditioned training procedure that learns hardware-aligned tile masks under a sparsity objective; an instruction set architecture whose instructions carry per-tile bitmask fields, allowing the hardware to skip masked tiles without software intervention; and a tiled inference accelerator with configurable parallelism, double-buffered memory, and INT8 datapath that natively supports sparse tile execution. We prototype on an AMD/Xilinx Alveo U50 FPGA and evaluate on a closed-loop visuomotor driving task in CARLA autonomous driving simulator. Task-conditional sparsity reduces FLOPs by 66-76% while maintaining driving quality. On-device latency decreases by 51-59%, from 9.12 ms to 3.74-4.44 ms (2.1-2.4x speedup), with energy per inference dropping from 263 to 108-128mJ.

Problem

• multi-task inference
• compute skipping
• task-conditional sparsity
• inference acceleration
• hardware efficiency

Innovation

• task-conditional sparsity
• compute skipping
• HW/SW co-design
• tiled inference accelerator
• multi-task inference