A Hardware-oriented Approach for Efficient Bayesian Inference Computation and Deployment

2026-07-20

AI Summary

This work addresses the high computational overhead of discrete Bayesian inference on resource-constrained edge devices. Targeting embedded GPUs, the authors propose a hardware-friendly tensor contraction optimization method that enhances inference efficiency while preserving exact numerical accuracy. The approach integrates memory layout restructuring, fused reduction strategies, sparse array representations, and tensor clustering, complemented by a machine learning–based autotuner to select the optimal implementation variant. Evaluated on an NVIDIA Jetson Orin AGX across 770 POMDP configurations, the method achieves typical speedups of 2–2.5× over the baseline, with peak acceleration reaching 5×.

Abstract

Bayesian inference provides a principled foundation for reasoning under uncertainty, but its computational cost hinders deployment on resource-constrained edge devices. In this paper, we present a hardware-oriented methodology for accelerating discrete Bayesian inference on commercial off-the-shelf embedded GPUs. We identify that the latency of a broad class of variational message-passing algorithms is dominated by tensor contractions. Our approach restructures the memory layout of these operations using two complementary merging strategies that produce compact, regularly-shaped primitives better suited for efficient GPU execution. We then introduce optional sparse array representations and a tensor-clustering scheme to reduce the memory footprint. We instantiate the methodology and produce optimized variants of three message-passing algorithms for Hidden Markov Models (HMMs), namely variational filtering, variational message passing, and marginal message passing. Furthermore, we complement this with a machine-learning-based autotuner that automatically selects the best-performing algorithmic variant for a given generative model specification. Benchmarked on an NVIDIA Jetson Orin AGX across 770 randomly sampled realistic Partially Observable Markov Decision Process (POMDP) configurations, our implementations achieve speedups of up to 5x, with typical gains of 2-2.5x, while producing numerically identical outputs to the baseline implementations.

Problem

• Bayesian inference
• edge devices
• tensor contractions
• embedded GPUs
• computational cost

Innovation

• hardware-oriented optimization
• tensor contraction
• variational message passing
• sparse tensor representation
• autotuning