inference optimization

This work addresses the high computational overhead of discrete Bayesian inference on resource-constrained edge devices. Targeting embedded GPUs, the authors propose a hardware-friendly tensor contraction optimization method that enhances inference efficiency while preserving exact numerical accuracy. The approach integrates memory layout restructuring, fused reduction strategies, sparse array representations, and tensor clustering, complemented by a machine learning–based autotuner to select the optimal implementation variant. Evaluated on an NVIDIA Jetson Orin AGX across 770 POMDP configurations, the method achieves typical speedups of 2–2.5× over the baseline, with peak acceleration reaching 5×.

Active Inference (AIF) suffers from high computational and memory overhead, hindering its deployment on resource-constrained real-time or embedded systems. To address this, we propose a hardware-efficient AIF computing architecture: leveraging the pymdp framework, we construct a sparse, unified computational graph that explicitly optimizes computation flow and memory access patterns while preserving model flexibility. This work presents the first customized hardware adaptation of AIF for edge devices. Experimental evaluation demonstrates substantial efficiency gains—over 2× latency reduction and up to 35% lower peak memory footprint—without compromising inference fidelity. Our core innovation lies in mapping AIF’s Bayesian inference process onto a sparse, statically schedulable computational graph, thereby jointly optimizing algorithmic accuracy and hardware execution efficiency. The proposed architecture provides a scalable, system-level solution for deploying lightweight active agents in practical edge scenarios.

Bayesian inference often faces a trade-off between computational efficiency and posterior accuracy, especially across multiple datasets. This paper proposes an adaptive hybrid inference workflow that—uniquely—integrates amortized variational inference (AVI) with Markov chain Monte Carlo (MCMC) in a dynamically coordinated manner. Leveraging principled posterior diagnostics, it constructs a Pareto frontier to enable automatic, optimal switching between AVI and MCMC. Computational reuse and scheduling optimization further boost inference throughput. The method unifies generative neural network modeling, MCMC refinement, and verifiable diagnostic mechanisms. Evaluated on tens of thousands of real and synthetic datasets, it achieves a 3.2× average speedup over standalone AVI or MCMC baselines, while preserving posterior fidelity—reducing KL divergence by 47% and increasing effective sample size (ESS) by 2.8×. This work delivers a scalable, efficient, and trustworthy solution for large-scale Bayesian inference.

This work addresses the severe memory capacity and bandwidth bottlenecks faced by generative AI during inference on resource-constrained devices, particularly in long-context and multimodal scenarios. The authors propose a hierarchical roofline performance model to systematically evaluate, for the first time, the bandwidth and latency requirements of high-bandwidth storage (HBS) in large-model long-context inference, establishing clear HBS performance thresholds necessary to achieve interactive throughput. For smaller models, they design an efficient memory utilization scheme leveraging bonded global buffer chips. Experimental results demonstrate that the proposed approaches substantially alleviate memory pressure and improve energy efficiency, offering a critical technical pathway for deploying generative AI at the edge.

This work addresses the challenge of deploying large language models on edge devices, which is hindered by a lack of systematic understanding of inference latency and energy efficiency scaling across heterogeneous hardware (CPU/GPU/NPU). The authors propose QEIL, a unified framework that, for the first time, uncovers stable power-law scaling behaviors of Transformer models with respect to latency, energy consumption, and task coverage. Leveraging these insights, QEIL introduces three composite metrics and a safety-aware intelligent scheduler to enable coordinated optimization across heterogeneous accelerators from diverse vendors. Through formal modeling, computational orchestration, thermal management, fault-tolerant execution, and hardware health monitoring, QEIL significantly improves energy efficiency, reduces latency, and expands task coverage across five model families—while preserving model accuracy and ensuring system safety.

This study addresses the limitations of dense large language models in long-chain reasoning, where fragmented KV caches and parallelization inefficiencies undermine traditional prefill extension strategies. Through systematic evaluation of dense and Mixture-of-Experts models ranging from 8B to 671B parameters on GPU clusters, the work uncovers critical performance bottlenecks: a sharp drop in data parallelism efficiency due to cache fragmentation, a nonlinear scaling inflection point in tensor parallelism around 32B parameters, and fundamental differences between sparse and dense architectures in interconnect bandwidth utilization and routing latency. Guided by extensive empirical analysis, the authors propose an architecture decision framework tailored to the “inference cliff” phenomenon, establishing design principles for next-generation LLM inference infrastructure that substantially improve resource utilization and throughput efficiency.

This work proposes a verifier-guided adaptive inference framework that overcomes the inefficiencies of static computation allocation in conventional test-time reasoning. By modeling inference as an iterative process of trajectory generation and selection, the method dynamically plans, selects tools, and adjusts computational strategies at each step, all under the unified guidance of a Process Reward Model (PRM). This approach achieves, for the first time, fine-grained, cross-iteration adaptive computation allocation based on PRM signals, transcending the limitations of fixed sampling and post-hoc reranking. Evaluated on challenging benchmarks—including MATH-500, AIME24, and AMO-Bench—the framework significantly outperforms existing test-time scaling methods, delivering higher accuracy while reducing wasteful generations and tool invocation overhead.

This study addresses the challenge of balancing performance and efficiency in large language model inference under constrained computational resources. Through systematic evaluation across model scales and configurations on the MMLU-Pro and BBH benchmarks, it investigates reasoning augmentation strategies—including self-consistency, self-refinement, multi-agent debate, and agent ensembling. Large-scale multi-configuration experiments reveal that multi-agent approaches consistently enhance performance on highly complex tasks. Under identical compute budgets, debate and agent ensembling outperform self-consistency by 1.3% and 2.7% in accuracy, respectively. With a 20× Chain-of-Thought (CoT) budget, reasoning augmentation boosts MMLU-Pro accuracy by up to 7.1%. The work further establishes practical guidelines for efficient agent ensembling and leverages Pareto frontiers to inform optimal strategy selection.

This work addresses the memory performance bottleneck arising from probabilistic computation in trustworthy artificial intelligence by proposing a unified memory analysis framework that jointly models deterministic memory accesses and stochastic sampling, treating the former as a limiting case of the latter. For the first time, it establishes a holistic perspective integrating data movement and stochasticity provisioning, revealing the operational mechanisms under entropy constraints and formulating memory-level evaluation criteria tailored for probabilistic AI. By synergistically combining analytical methods from probabilistic computing and memory architecture, and incorporating assessments of distribution programmability, parallel compatibility, and robustness to hardware non-idealities, the study clarifies the limitations of conventional architectures and outlines a scalable pathway toward probabilistic in-memory computing hardware, thereby laying a theoretical and design foundation for efficient and trustworthy AI systems.