Multimodal sensor-based semantic 3D mapping for a large-scale environment

Jongmin Jeong, Tae Sung Yoon, Jin Bae Park

Research output: Contribution to journalArticlepeer-review

23 Citations (Scopus)


Semantic 3D mapping is one of the most important fields in robotics, and has been used in many applications, such as robot navigation, surveillance, and virtual reality. In general, semantic 3D mapping is mainly composed of 3D reconstruction and semantic segmentation. As these technologies evolve, there has been great progress in semantic 3D mapping in recent years. Furthermore, the number of robotic applications requiring semantic information in 3D mapping to perform high-level tasks has increased, and many studies on semantic 3D mapping have been published. Existing methods use a camera for both 3D reconstruction and semantic segmentation. However, this is not suitable for large-scale environments and has the disadvantage of high computational complexity. To address this problem, we propose a multimodal sensor-based semantic 3D mapping system using a 3D Lidar combined with a camera. In this study, the odometry is obtained by high-precision global positioning system (GPS) and inertial measurement unit (IMU), and it is estimated by iterative closest point (ICP) when a GPS signal is weak. Then, we use the latest 2D convolutional neural network (CNN) for semantic segmentation. To build a semantic 3D map, we integrate the 3D map with semantic information by using coordinate transformation and Bayes’ update scheme. In order to improve the semantic 3D map, we propose a 3D refinement process to correct wrongly segmented voxels and remove traces of moving vehicles in the 3D map. Through experiments on challenging sequences, we demonstrate that our method outperforms state-of-the-art methods in terms of accuracy and intersection over union (IoU). Thus, our method can be used for various applications that require semantic information in 3D map.

Original languageEnglish
Pages (from-to)1-10
Number of pages10
JournalExpert Systems with Applications
Publication statusPublished - 2018 Sept 1

Bibliographical note

Funding Information:
This research was supported by a National Research Foundation of Korea (NRF) grant funded by the Korea Government (MSIP) (No. NRF-2015R1A2A2A01007545 ), and was partially supported by Microsoft Research.

Publisher Copyright:
© 2018 Elsevier Ltd

All Science Journal Classification (ASJC) codes

  • Engineering(all)
  • Computer Science Applications
  • Artificial Intelligence


Dive into the research topics of 'Multimodal sensor-based semantic 3D mapping for a large-scale environment'. Together they form a unique fingerprint.

Cite this