Distributing workloads that cannot be handled by a single edge device across multiple edge devices is a promising solution that minimizes the inference latency of deep learning applications by exploiting model parallelism. Several prior solutions have been proposed to partition target models efficiently, but most studies have focused on finding the optimal fused layer configurations, which minimize the data-transfer overhead between layers. However, as recent deep learning network models have become more complex and the ability to deploy them quickly has become a key challenge, the search for the best fused layer configurations of target models has become a major requirement. To solve this problem, we propose a lightweight model partitioning framework called Legion to find the optimal fused layer configurations with minimal profiling execution trials. By finding the optimal configurations using cost matrix construction and wild card selection, the experimental results showed that Legion achieved a similar performance to the full configuration search at a fraction of the search time. Moreover, Legion performed effectively even on a group of heterogeneous target devices by introducing a per-device cost-related matrix construction. With three popular networks, Legion shows only 3.4% performance loss as compared to a full searching scheme (FSS), on various different device configurations consisting of up to six heterogeneous devices, and minimizes the profiling overhead by 48.7× on average.
|Title of host publication||Proceedings - 2021 IEEE 39th International Conference on Computer Design, ICCD 2021|
|Publisher||Institute of Electrical and Electronics Engineers Inc.|
|Number of pages||8|
|Publication status||Published - 2021|
|Event||39th IEEE International Conference on Computer Design, ICCD 2021 - Virtual, Online, United States|
Duration: 2021 Oct 24 → 2021 Oct 27
|Name||Proceedings - IEEE International Conference on Computer Design: VLSI in Computers and Processors|
|Conference||39th IEEE International Conference on Computer Design, ICCD 2021|
|Period||21/10/24 → 21/10/27|
Bibliographical noteFunding Information:
ACKNOWLEDGMENT This work was supported by Samsung Research Funding & Incubation Center of Samsung Electronics under Project Number SRFC-IT1901-03, and by the research fund of Hanyang University (HY-2017). Yongjun Park is the corresponding author.
© 2021 IEEE.
All Science Journal Classification (ASJC) codes
- Hardware and Architecture
- Electrical and Electronic Engineering