DyCE: Dynamically Configurable Exiting for deep learning compression and real-time scaling

2024-03-04

Future generations computer systems

AI Summary

To address the challenge of simultaneously achieving real-time inference and energy efficiency for deep learning models on edge devices, this paper proposes a hardware-aware, fine-grained, dynamically configurable early-exit mechanism. It enables runtime adaptation—selecting optimal exit layers based on instantaneous resource conditions (e.g., latency, power consumption)—to realize deployment-free, real-time model compression and scaling. Unlike static models or monolithic dynamic inference approaches, our method introduces a multi-exit network architecture integrating gradient-sensitivity-driven exit placement, lightweight gating controllers, and an online resource-feedback scheduling algorithm. This establishes the first dynamic configuration paradigm jointly optimizing inference latency, accuracy, and energy consumption. Evaluated on ImageNet, our approach achieves up to 3.2× speedup and 58% energy reduction, with accuracy degradation under 0.8%, while enabling millisecond-level exit-policy switching.

Technology Category

Machine Learning: Learning on the Edge & Model CompressionComputer Vision: Learning & Optimization for CVSearch and Optimization: Learning to Search

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsSystems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environmentsEconomics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Problem

• Dynamic model adaptation for varying sample complexity
• Runtime performance-complexity trade-off adjustment without redeployment
• Generalizable compression and scaling via exit networks

Innovation

• Dynamic exiting via small intermediate exit networks
• Decouples design for easy adaptation to new models
• Real-time performance-complexity trade-off adjustment