Unsupervised Deep Image Fusion with Structure Tensor Representations

Hyungjoo Jung, Youngjung Kim, Hyunsung Jang, Namkoo Ha, Kwanghoon Sohn

Research output: Contribution to journalArticlepeer-review

100 Citations (Scopus)


Convolutional neural networks (CNNs) have facilitated substantial progress on various problems in computer vision and image processing. However, applying them to image fusion has remained challenging due to the lack of the labelled data for supervised learning. This paper introduces a deep image fusion network (DIF-Net), an unsupervised deep learning framework for image fusion. The DIF-Net parameterizes the entire processes of image fusion, comprising of feature extraction, feature fusion, and image reconstruction, using a CNN. The purpose of DIF-Net is to generate an output image which has an identical contrast to high-dimensional input images. To realize this, we propose an unsupervised loss function using the structure tensor representation of the multi-channel image contrasts. Different from traditional fusion methods that involve time-consuming optimization or iterative procedures to obtain the results, our loss function is minimized by a stochastic deep learning solver with large-scale examples. Consequently, the proposed method can produce fused images that preserve source image details through a single forward network trained without reference ground-truth labels. The proposed method has broad applicability to various image fusion problems, including multi-spectral, multi-focus, and multi-exposure image fusions. Quantitative and qualitative evaluations show that the proposed technique outperforms existing state-of-the-art approaches for various applications.

Original languageEnglish
Article number8962327
Pages (from-to)3845-3858
Number of pages14
JournalIEEE Transactions on Image Processing
Publication statusPublished - 2020

Bibliographical note

Publisher Copyright:
© 1992-2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Unsupervised Deep Image Fusion with Structure Tensor Representations'. Together they form a unique fingerprint.

Cite this