Dynamic thresholding for learning sparse neural networks

Jin Woo Park, Jong Seok Lee

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Citation (Scopus)

Abstract

This paper proposes a method called Dynamic Thresholding, which can dynamically adjust the size of deep neural networks by removing redundant weights during training. The key idea is to learn the pruning threshold values applied for weight removal, instead of fixing them manually. We approximate a discontinuous pruning function with a differentiable form involving the thresholds, which can be optimized via the gradient descent learning procedure. While previous sparsity-promoting methods perform pruning with manually determined thresholds, our method can directly obtain a sparse network at each training iteration and thus does not need a trial-and-error process to choose proper threshold values. We examine the performance of the proposed method on the image classification tasks including MNIST, CIFAR10, and ImageNet. It is demonstrated that our method achieves competitive results with existing methods and, at the same time, requires smaller numbers of training iterations in comparison to other approaches based on train-prune-retrain cycles.

Original languageEnglish
Title of host publicationECAI 2020 - 24th European Conference on Artificial Intelligence, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020 - Proceedings
EditorsGiuseppe De Giacomo, Alejandro Catala, Bistra Dilkina, Michela Milano, Senen Barro, Alberto Bugarin, Jerome Lang
PublisherIOS Press BV
Pages1403-1410
Number of pages8
ISBN (Electronic)9781643681009
DOIs
Publication statusPublished - 2020 Aug 24
Event24th European Conference on Artificial Intelligence, ECAI 2020, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020 - Santiago de Compostela, Online, Spain
Duration: 2020 Aug 292020 Sept 8

Publication series

NameFrontiers in Artificial Intelligence and Applications
Volume325
ISSN (Print)0922-6389
ISSN (Electronic)1879-8314

Conference

Conference24th European Conference on Artificial Intelligence, ECAI 2020, including 10th Conference on Prestigious Applications of Artificial Intelligence, PAIS 2020
Country/TerritorySpain
CitySantiago de Compostela, Online
Period20/8/2920/9/8

Bibliographical note

Publisher Copyright:
© 2020 The authors and IOS Press.

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence

Fingerprint

Dive into the research topics of 'Dynamic thresholding for learning sparse neural networks'. Together they form a unique fingerprint.

Cite this