Stacking-Based Deep Neural Network: Deep Analytic Network for Pattern Classification

Cheng Yaw Low, Jaewoo Park, Andrew Beng Jin Teoh

Research output: Contribution to journalArticlepeer-review

18 Citations (Scopus)


Stacking-based deep neural network (S-DNN) is aggregated with pluralities of basic learning modules, one after another, to synthesize a deep neural network (DNN) alternative for pattern classification. Contrary to the DNNs trained from end to end by backpropagation (BP), each S-DNN layer, that is, a self-learnable module, is to be trained decisively and independently without BP intervention. In this paper, a ridge regression-based S-DNN, dubbed deep analytic network (DAN), along with its kernelization (K-DAN), are devised for multilayer feature relearning from the pre-extracted baseline features and the structured features. Our theoretical formulation demonstrates that DAN/K-DAN relearn by perturbing the intra/interclass variations, apart from diminishing the prediction errors. We scrutinize the DAN/K-DAN performance for pattern classification on datasets of varying domains - faces, handwritten digits, generic objects, to name a few. Unlike the typical BP-optimized DNNs to be trained from gigantic datasets by GPU, we reveal that DAN/K-DAN are trainable using only CPU even for small-scale training sets. Our experimental results show that DAN/K-DAN outperform the present S-DNNs and also the BP-trained DNNs, including multiplayer perceptron, deep belief network, etc., without data augmentation applied.

Original languageEnglish
Article number8694849
Pages (from-to)5021-5034
Number of pages14
JournalIEEE Transactions on Cybernetics
Issue number12
Publication statusPublished - 2020 Dec

Bibliographical note

Funding Information:
Manuscript received August 28, 2018; revised November 17, 2018; accepted March 26, 2019. Date of publication April 22, 2019; date of current version December 3, 2020. This work was supported in part by the National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIP) under Grant NRF-2019R1A2C1003306. This paper was recommended by Associate Editor Y. Zhang. (Corresponding author: Andrew Beng-Jin Teoh.) C.-Y. Low was with the School of Electrical and Electronic Engineering, College of Engineering, Yonsei University, Seoul 03722, South Korea. He is now with the Faculty of Information Science and Technology, Multimedia University, Melaka 75450, Malaysia (e-mail:

Publisher Copyright:
© 2013 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Control and Systems Engineering
  • Information Systems
  • Human-Computer Interaction
  • Computer Science Applications
  • Electrical and Electronic Engineering


Dive into the research topics of 'Stacking-Based Deep Neural Network: Deep Analytic Network for Pattern Classification'. Together they form a unique fingerprint.

Cite this