RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval

Hung Yu Tseng; Hsin Ying Lee; Lu Jiang; Ming Hsuan Yang; Weilong Yang

doi:10.1007/978-3-030-58598-3_15

RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval

Hung Yu Tseng, Hsin Ying Lee, Lu Jiang, Ming Hsuan Yang, Weilong Yang

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

7 Citations (Scopus)

Abstract

Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing. In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module. With the differentiable retrieval module, we can (1) make the entire pipeline end-to-end trainable, enabling the learning of better feature embedding for retrieval; (2) encourage the selection of mutually compatible patches with additional objective functions. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually compatible.

Original language	English
Title of host publication	Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings
Editors	Andrea Vedaldi, Horst Bischof, Thomas Brox, Jan-Michael Frahm
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	242-257
Number of pages	16
ISBN (Print)	9783030585976
DOIs	https://doi.org/10.1007/978-3-030-58598-3_15
Publication status	Published - 2020
Event	16th European Conference on Computer Vision, ECCV 2020 - Glasgow, United Kingdom Duration: 2020 Aug 23 → 2020 Aug 28

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12353 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	16th European Conference on Computer Vision, ECCV 2020
Country/Territory	United Kingdom
City	Glasgow
Period	20/8/23 → 20/8/28

Bibliographical note

Funding Information:
supported in part by the NSF CAREER Grant

Funding Information:
This work is supported in part by the NSF CAREER Grant #1149783.

Publisher Copyright:
© 2020, Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-030-58598-3_15

Cite this

Tseng, H. Y., Lee, H. Y., Jiang, L., Yang, M. H., & Yang, W. (2020). RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval. In A. Vedaldi, H. Bischof, T. Brox, & J-M. Frahm (Eds.), Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings (pp. 242-257). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12353 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-58598-3_15

Tseng, Hung Yu ; Lee, Hsin Ying ; Jiang, Lu et al. / RetrieveGAN : Image Synthesis via Differentiable Patch Retrieval. Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings. editor / Andrea Vedaldi ; Horst Bischof ; Thomas Brox ; Jan-Michael Frahm. Springer Science and Business Media Deutschland GmbH, 2020. pp. 242-257 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{68eaab569e2f4c9d82357591954c0683,

title = "RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval",

abstract = "Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing. In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module. With the differentiable retrieval module, we can (1) make the entire pipeline end-to-end trainable, enabling the learning of better feature embedding for retrieval; (2) encourage the selection of mutually compatible patches with additional objective functions. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually compatible.",

author = "Tseng, {Hung Yu} and Lee, {Hsin Ying} and Lu Jiang and Yang, {Ming Hsuan} and Weilong Yang",

note = "Funding Information: supported in part by the NSF CAREER Grant Funding Information: This work is supported in part by the NSF CAREER Grant #1149783. Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 16th European Conference on Computer Vision, ECCV 2020 ; Conference date: 23-08-2020 Through 28-08-2020",

year = "2020",

doi = "10.1007/978-3-030-58598-3_15",

language = "English",

isbn = "9783030585976",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "242--257",

editor = "Andrea Vedaldi and Horst Bischof and Thomas Brox and Jan-Michael Frahm",

booktitle = "Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings",

address = "Germany",

}

Tseng, HY, Lee, HY, Jiang, L, Yang, MH & Yang, W 2020, RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval. in A Vedaldi, H Bischof, T Brox & J-M Frahm (eds), Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12353 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 242-257, 16th European Conference on Computer Vision, ECCV 2020, Glasgow, United Kingdom, 20/8/23. https://doi.org/10.1007/978-3-030-58598-3_15

RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval. / Tseng, Hung Yu; Lee, Hsin Ying; Jiang, Lu et al.
Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings. ed. / Andrea Vedaldi; Horst Bischof; Thomas Brox; Jan-Michael Frahm. Springer Science and Business Media Deutschland GmbH, 2020. p. 242-257 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12353 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - RetrieveGAN

T2 - 16th European Conference on Computer Vision, ECCV 2020

AU - Tseng, Hung Yu

AU - Lee, Hsin Ying

AU - Jiang, Lu

AU - Yang, Ming Hsuan

AU - Yang, Weilong

N1 - Funding Information: supported in part by the NSF CAREER Grant Funding Information: This work is supported in part by the NSF CAREER Grant #1149783. Publisher Copyright: © 2020, Springer Nature Switzerland AG.

PY - 2020

Y1 - 2020

N2 - Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing. In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module. With the differentiable retrieval module, we can (1) make the entire pipeline end-to-end trainable, enabling the learning of better feature embedding for retrieval; (2) encourage the selection of mutually compatible patches with additional objective functions. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually compatible.

AB - Image generation from scene description is a cornerstone technique for the controlled generation, which is beneficial to applications such as content creation and image editing. In this work, we aim to synthesize images from scene description with retrieved patches as reference. We propose a differentiable retrieval module. With the differentiable retrieval module, we can (1) make the entire pipeline end-to-end trainable, enabling the learning of better feature embedding for retrieval; (2) encourage the selection of mutually compatible patches with additional objective functions. We conduct extensive quantitative and qualitative experiments to demonstrate that the proposed method can generate realistic and diverse images, where the retrieved patches are reasonable and mutually compatible.

UR - http://www.scopus.com/inward/record.url?scp=85093070388&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85093070388&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-58598-3_15

DO - 10.1007/978-3-030-58598-3_15

M3 - Conference contribution

AN - SCOPUS:85093070388

SN - 9783030585976

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 242

EP - 257

BT - Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings

A2 - Vedaldi, Andrea

A2 - Bischof, Horst

A2 - Brox, Thomas

A2 - Frahm, Jan-Michael

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 23 August 2020 through 28 August 2020

ER -

Tseng HY, Lee HY, Jiang L, Yang MH, Yang W. RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval. In Vedaldi A, Bischof H, Brox T, Frahm J-M, editors, Computer Vision – ECCV 2020 - 16th European Conference, 2020, Proceedings. Springer Science and Business Media Deutschland GmbH. 2020. p. 242-257. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-58598-3_15

RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this