Fulcrum: Optimizing Concurrent DNN Training and Inferencing on Edge Accelerators

2025-09-24

AI Summary

Edge accelerators (e.g., Jetson) lack GPU sharing mechanisms, exhibit numerous power states, and struggle to coordinate DNN training and inference efficiently. To address this, we propose an intelligent dynamic time-slicing scheduling framework. Our method jointly optimizes power modes and batch sizes by integrating multi-dimensional gradient descent (GMD) with active learning (ALS), achieving rapid convergence to Pareto-optimal configurations under low profiling overhead. It enables cross-workload policy reuse and maximizes training throughput under joint latency and power constraints. Evaluated across a configuration space of 273,000 possibilities, our approach satisfies both constraints in over 97% of scenarios and attains, on average, 93% of the theoretical maximum throughput—significantly outperforming state-of-the-art baselines.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Learning to Search
• Planning, Routing, and Scheduling: Planning/Scheduling and Learning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The proliferation of GPU accelerated edge devices like Nvidia Jetsons and the rise in privacy concerns are placing an emphasis on concurrent DNN training and inferencing on edge devices. Inference and training have different computing and QoS goals. But edge accelerators like Jetson do not support native GPU sharing and expose 1000s of power modes. This requires careful time-sharing of concurrent workloads to meet power--performance goals, while limiting costly profiling. In this paper, we design an intelligent time-slicing approach for concurrent DNN training and inferencing on Jetsons. We formulate an optimization problem to interleave training and inferencing minibatches, and decide the device power mode and inference minibatch size, while maximizing the training throughput and staying within latency and power budgets, with modest profiling costs. We propose GMD, an efficient multi-dimensional gradient descent search which profiles just $15$ power modes; and ALS, an Active Learning technique which identifies reusable Pareto-optimal power modes, but profiles $50$--$150$ power modes. We evaluate these within our Fulcrum scheduler for $273,000+$ configurations across $15$ DNN workloads. We also evaluate our strategies on dynamic arrival inference and concurrent inferences. ALS and GMD outperform simpler and more complex baselines with larger-scale profiling. Their solutions satisfy the latency and power budget for $>97%$ of our runs, and on average are within $7%$ of the optimal throughput.

Problem

• Optimizing concurrent DNN training and inferencing on edge GPU accelerators
• Managing power-performance trade-offs without native GPU sharing support
• Minimizing costly profiling while meeting latency and power constraints

Innovation

• Intelligent time-slicing for concurrent DNN workloads
• GMD gradient descent search with minimal power mode profiling
• Active Learning technique identifying reusable Pareto-optimal configurations