A deep neural network ensemble of multimodal signals for classifying excavator operations

Jin Young Kim, Sung Bae Cho

Research output: Contribution to journalArticlepeer-review

4 Citations (Scopus)


The prognostics and health management (PHM) aims to provide a comprehensive solution for equipment health care. Classifying the operation mode of excavator, one of the tasks in the PHM, is important to evaluate the remaining useful lifetime. Several studies have been conducted to classify the operations with either video or sensor data, but they have several limitations to use only one type of data. A model trained with sensor data cannot classify the similar operations such as “digging” and “ditch digging”, whereas a model with video data is vulnerable to surrounding condition like weather. In this paper, to overcome these shortcomings, we propose a deep neural network ensemble called FusionNet that classifies the operations of excavator. Two models are trained with sensor data and video frames respectively, where the feature extractors are transferred to the FusionNet. The proposed network ensemble performs a flexible and well-optimized classification by automatically calculating weights according to the extracted feature vectors and combining them. To verify the proposed model, several experiments are conducted with the real-world data. The proposed model achieves the accuracy of 99.17% which outperforms the conventional methods. We also confirm that the proposed model can address the shortcomings of using only one type of data and maximize the benefits through the automatic weighting of extracted features.

Original languageEnglish
Pages (from-to)290-299
Number of pages10
Publication statusPublished - 2022 Jan 22

Bibliographical note

Funding Information:
This work was supported by Institute of Information & Communications Technology Planning & Evaluation (IITP) grant funded by the Korean government (MSIT) (No. 2020-0-01361, Artificial Intelligence Graduate School Program (Yonsei University)) and Electronics and Telecommunications Research Institute (ETRI) grant funded by the Korean government (21ZS1100, Core Technology Research for Self-Improving Integrated Artificial Intelligence System). J. Y. Kim has been supported by NRF ( National Research Foundation of Korea ) grant funded by the Korean government (NRF-2019-Fostering Core Leaders of the Future Basic Science Program/Global Ph.D. Fellowship Program).

Publisher Copyright:
© 2021 Elsevier B.V.

All Science Journal Classification (ASJC) codes

  • Computer Science Applications
  • Cognitive Neuroscience
  • Artificial Intelligence


Dive into the research topics of 'A deep neural network ensemble of multimodal signals for classifying excavator operations'. Together they form a unique fingerprint.

Cite this