Physics-Based Generative Adversarial Models for Image Restoration and beyond

Jinshan Pan, Jiangxin Dong, Yang Liu, Jiawei Zhang, Jimmy Ren, Jinhui Tang, Yu Wing Tai, Ming Hsuan Yang

Research output: Contribution to journalArticlepeer-review

64 Citations (Scopus)


We present an algorithm to directly solve numerous image restoration problems (e.g., image deblurring, image dehazing, and image deraining). These problems are ill-posed, and the common assumptions for existing methods are usually based on heuristic image priors. In this paper, we show that these problems can be solved by generative models with adversarial learning. However, a straightforward formulation based on a straightforward generative adversarial network (GAN) does not perform well in these tasks, and some structures of the estimated images are usually not preserved well. Motivated by an interesting observation that the estimated results should be consistent with the observed inputs under the physics models, we propose an algorithm that guides the estimation process of a specific task within the GAN framework. The proposed model is trained in an end-To-end fashion and can be applied to a variety of image restoration and low-level vision problems. Extensive experiments demonstrate that the proposed method performs favorably against state-of-The-Art algorithms.

Original languageEnglish
Article number8968618
Pages (from-to)2449-2462
Number of pages14
JournalIEEE transactions on pattern analysis and machine intelligence
Issue number7
Publication statusPublished - 2021 Jul 1

Bibliographical note

Funding Information:
This work was supported in part by the National Natural Science Foundation of China (Nos. 61922043, 61872421, 61732007), the Natural Science Foundation of Jiangsu Province (No. BK20180471), and National Science Foundation CAREER (No. 1149783).

Publisher Copyright:
© 1979-2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'Physics-Based Generative Adversarial Models for Image Restoration and beyond'. Together they form a unique fingerprint.

Cite this