GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent

Heesu Kim; Hanmin Park; Taehyun Kim; Kwanheum Cho; Eojin Lee; Soojung Ryu; Hyuk Jae Lee; Kiyoung Choi; Jinho Lee

doi:10.1109/HPCA51647.2021.00030

GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent

Heesu Kim, Hanmin Park, Taehyun Kim, Kwanheum Cho, Eojin Lee, Soojung Ryu, Hyuk Jae Lee, Kiyoung Choi, Jinho Lee

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

14 Citations (Scopus)

Abstract

In this paper, we present GradPIM, a processingin-memory architecture which accelerates parameter updates of deep neural networks training. As one of processing-in-memory techniques that could be realized in the near future, we propose an incremental, simple architectural design that does not invade the existing memory protocol. Extending DDR4 SDRAM to utilize bank-group parallelism makes our operation designs in processing-in-memory (PIM) module efficient in terms of hardware cost and performance. Our experimental results show that the proposed architecture can improve the performance of DNN training and greatly reduce memory bandwidth requirement while posing only a minimal amount of overhead to the protocol and DRAM area.

Original language	English
Title of host publication	Proceeding - 27th IEEE International Symposium on High Performance Computer Architecture, HPCA 2021
Publisher	IEEE Computer Society
Pages	249-262
Number of pages	14
ISBN (Electronic)	9780738123370
DOIs	https://doi.org/10.1109/HPCA51647.2021.00030
Publication status	Published - 2021 Feb
Event	27th Annual IEEE International Symposium on High Performance Computer Architecture, HPCA 2021 - Virtual, Seoul, Korea, Republic of Duration: 2021 Feb 27 → 2021 Mar 1

Publication series

Name	Proceedings - International Symposium on High-Performance Computer Architecture
Volume	2021-February
ISSN (Print)	1530-0897

Conference

Conference	27th Annual IEEE International Symposium on High Performance Computer Architecture, HPCA 2021
Country/Territory	Korea, Republic of
City	Virtual, Seoul
Period	21/2/27 → 21/3/1

Bibliographical note

Publisher Copyright:
© 2021 IEEE.

All Science Journal Classification (ASJC) codes

Hardware and Architecture

Access to Document

10.1109/HPCA51647.2021.00030

Cite this

Kim, H., Park, H., Kim, T., Cho, K., Lee, E., Ryu, S., Lee, H. J., Choi, K., & Lee, J. (2021). GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent. In Proceeding - 27th IEEE International Symposium on High Performance Computer Architecture, HPCA 2021 (pp. 249-262). Article 9407195 (Proceedings - International Symposium on High-Performance Computer Architecture; Vol. 2021-February). IEEE Computer Society. https://doi.org/10.1109/HPCA51647.2021.00030

@inproceedings{2c1097be5b584c5f93bfe685986216c8,

title = "GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent",

abstract = "In this paper, we present GradPIM, a processingin-memory architecture which accelerates parameter updates of deep neural networks training. As one of processing-in-memory techniques that could be realized in the near future, we propose an incremental, simple architectural design that does not invade the existing memory protocol. Extending DDR4 SDRAM to utilize bank-group parallelism makes our operation designs in processing-in-memory (PIM) module efficient in terms of hardware cost and performance. Our experimental results show that the proposed architecture can improve the performance of DNN training and greatly reduce memory bandwidth requirement while posing only a minimal amount of overhead to the protocol and DRAM area.",

author = "Heesu Kim and Hanmin Park and Taehyun Kim and Kwanheum Cho and Eojin Lee and Soojung Ryu and Lee, {Hyuk Jae} and Kiyoung Choi and Jinho Lee",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 27th Annual IEEE International Symposium on High Performance Computer Architecture, HPCA 2021 ; Conference date: 27-02-2021 Through 01-03-2021",

year = "2021",

month = feb,

doi = "10.1109/HPCA51647.2021.00030",

language = "English",

series = "Proceedings - International Symposium on High-Performance Computer Architecture",

publisher = "IEEE Computer Society",

pages = "249--262",

booktitle = "Proceeding - 27th IEEE International Symposium on High Performance Computer Architecture, HPCA 2021",

address = "United States",

}

Kim, H, Park, H, Kim, T, Cho, K, Lee, E, Ryu, S, Lee, HJ, Choi, K & Lee, J 2021, GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent. in Proceeding - 27th IEEE International Symposium on High Performance Computer Architecture, HPCA 2021., 9407195, Proceedings - International Symposium on High-Performance Computer Architecture, vol. 2021-February, IEEE Computer Society, pp. 249-262, 27th Annual IEEE International Symposium on High Performance Computer Architecture, HPCA 2021, Virtual, Seoul, Korea, Republic of, 21/2/27. https://doi.org/10.1109/HPCA51647.2021.00030

GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent. / Kim, Heesu; Park, Hanmin; Kim, Taehyun et al.
Proceeding - 27th IEEE International Symposium on High Performance Computer Architecture, HPCA 2021. IEEE Computer Society, 2021. p. 249-262 9407195 (Proceedings - International Symposium on High-Performance Computer Architecture; Vol. 2021-February).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - GradPIM

T2 - 27th Annual IEEE International Symposium on High Performance Computer Architecture, HPCA 2021

AU - Kim, Heesu

AU - Park, Hanmin

AU - Kim, Taehyun

AU - Cho, Kwanheum

AU - Lee, Eojin

AU - Ryu, Soojung

AU - Lee, Hyuk Jae

AU - Choi, Kiyoung

AU - Lee, Jinho

PY - 2021/2

Y1 - 2021/2

N2 - In this paper, we present GradPIM, a processingin-memory architecture which accelerates parameter updates of deep neural networks training. As one of processing-in-memory techniques that could be realized in the near future, we propose an incremental, simple architectural design that does not invade the existing memory protocol. Extending DDR4 SDRAM to utilize bank-group parallelism makes our operation designs in processing-in-memory (PIM) module efficient in terms of hardware cost and performance. Our experimental results show that the proposed architecture can improve the performance of DNN training and greatly reduce memory bandwidth requirement while posing only a minimal amount of overhead to the protocol and DRAM area.

AB - In this paper, we present GradPIM, a processingin-memory architecture which accelerates parameter updates of deep neural networks training. As one of processing-in-memory techniques that could be realized in the near future, we propose an incremental, simple architectural design that does not invade the existing memory protocol. Extending DDR4 SDRAM to utilize bank-group parallelism makes our operation designs in processing-in-memory (PIM) module efficient in terms of hardware cost and performance. Our experimental results show that the proposed architecture can improve the performance of DNN training and greatly reduce memory bandwidth requirement while posing only a minimal amount of overhead to the protocol and DRAM area.

UR - http://www.scopus.com/inward/record.url?scp=85104983900&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85104983900&partnerID=8YFLogxK

U2 - 10.1109/HPCA51647.2021.00030

DO - 10.1109/HPCA51647.2021.00030

M3 - Conference contribution

AN - SCOPUS:85104983900

T3 - Proceedings - International Symposium on High-Performance Computer Architecture

SP - 249

EP - 262

BT - Proceeding - 27th IEEE International Symposium on High Performance Computer Architecture, HPCA 2021

PB - IEEE Computer Society

Y2 - 27 February 2021 through 1 March 2021

ER -

Kim H, Park H, Kim T, Cho K, Lee E, Ryu S et al. GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent. In Proceeding - 27th IEEE International Symposium on High Performance Computer Architecture, HPCA 2021. IEEE Computer Society. 2021. p. 249-262. 9407195. (Proceedings - International Symposium on High-Performance Computer Architecture). doi: 10.1109/HPCA51647.2021.00030

GradPIM: A Practical Processing-in-DRAM Architecture for Gradient Descent

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this