Deep Semantic Matching with Foreground Detection and Cycle-Consistency

Yun Chun Chen; Po Hsiang Huang; Li Yu Yu; Jia Bin Huang; Ming Hsuan Yang; Yen Yu Lin

doi:10.1007/978-3-030-20893-6_22

Deep Semantic Matching with Foreground Detection and Cycle-Consistency

Yun Chun Chen, Po Hsiang Huang, Li Yu Yu, Jia Bin Huang, Ming Hsuan Yang, Yen Yu Lin

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

10 Citations (Scopus)

Abstract

Establishing dense semantic correspondences between object instances remains a challenging problem due to background clutter, significant scale and pose differences, and large intra-class variations. In this paper, we present an end-to-end trainable network for learning semantic correspondences using only matching image pairs without manual keypoint correspondence annotations. To facilitate network training with this weaker form of supervision, we (1) explicitly estimate the foreground regions to suppress the effect of background clutter and (2) develop cycle-consistent losses to enforce the predicted transformations across multiple images to be geometrically plausible and consistent. We train the proposed model using the PF-PASCAL dataset and evaluate the performance on the PF-PASCAL, PF-WILLOW, and TSS datasets. Extensive experimental results show that the proposed approach achieves favorably performance compared to the state-of-the-art. The code and model will be available at https://yunchunchen.github.io/WeakMatchNet/.

Original language	English
Title of host publication	Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers
Editors	C.V. Jawahar, Konrad Schindler, Hongdong Li, Greg Mori
Publisher	Springer Verlag
Pages	347-362
Number of pages	16
ISBN (Print)	9783030208929
DOIs	https://doi.org/10.1007/978-3-030-20893-6_22
Publication status	Published - 2019
Event	14th Asian Conference on Computer Vision, ACCV 2018 - Perth, Australia Duration: 2018 Dec 2 → 2018 Dec 6

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11363 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	14th Asian Conference on Computer Vision, ACCV 2018
Country/Territory	Australia
City	Perth
Period	18/12/2 → 18/12/6

Bibliographical note

Publisher Copyright:
© 2019, Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-030-20893-6_22

Cite this

Chen, Y. C., Huang, P. H., Yu, L. Y., Huang, J. B., Yang, M. H., & Lin, Y. Y. (2019). Deep Semantic Matching with Foreground Detection and Cycle-Consistency. In C. V. Jawahar, K. Schindler, H. Li, & G. Mori (Eds.), Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers (pp. 347-362). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11363 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-20893-6_22

Chen, Yun Chun ; Huang, Po Hsiang ; Yu, Li Yu et al. / Deep Semantic Matching with Foreground Detection and Cycle-Consistency. Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. editor / C.V. Jawahar ; Konrad Schindler ; Hongdong Li ; Greg Mori. Springer Verlag, 2019. pp. 347-362 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{7360c16f4b5e4dbf890aa7bd6c042c70,

title = "Deep Semantic Matching with Foreground Detection and Cycle-Consistency",

abstract = "Establishing dense semantic correspondences between object instances remains a challenging problem due to background clutter, significant scale and pose differences, and large intra-class variations. In this paper, we present an end-to-end trainable network for learning semantic correspondences using only matching image pairs without manual keypoint correspondence annotations. To facilitate network training with this weaker form of supervision, we (1) explicitly estimate the foreground regions to suppress the effect of background clutter and (2) develop cycle-consistent losses to enforce the predicted transformations across multiple images to be geometrically plausible and consistent. We train the proposed model using the PF-PASCAL dataset and evaluate the performance on the PF-PASCAL, PF-WILLOW, and TSS datasets. Extensive experimental results show that the proposed approach achieves favorably performance compared to the state-of-the-art. The code and model will be available at https://yunchunchen.github.io/WeakMatchNet/.",

author = "Chen, {Yun Chun} and Huang, {Po Hsiang} and Yu, {Li Yu} and Huang, {Jia Bin} and Yang, {Ming Hsuan} and Lin, {Yen Yu}",

note = "Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 14th Asian Conference on Computer Vision, ACCV 2018 ; Conference date: 02-12-2018 Through 06-12-2018",

year = "2019",

doi = "10.1007/978-3-030-20893-6_22",

language = "English",

isbn = "9783030208929",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "347--362",

editor = "C.V. Jawahar and Konrad Schindler and Hongdong Li and Greg Mori",

booktitle = "Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers",

address = "Germany",

}

Chen, YC, Huang, PH, Yu, LY, Huang, JB, Yang, MH & Lin, YY 2019, Deep Semantic Matching with Foreground Detection and Cycle-Consistency. in CV Jawahar, K Schindler, H Li & G Mori (eds), Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11363 LNCS, Springer Verlag, pp. 347-362, 14th Asian Conference on Computer Vision, ACCV 2018, Perth, Australia, 18/12/2. https://doi.org/10.1007/978-3-030-20893-6_22

Deep Semantic Matching with Foreground Detection and Cycle-Consistency. / Chen, Yun Chun; Huang, Po Hsiang; Yu, Li Yu et al.
Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. ed. / C.V. Jawahar; Konrad Schindler; Hongdong Li; Greg Mori. Springer Verlag, 2019. p. 347-362 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11363 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Deep Semantic Matching with Foreground Detection and Cycle-Consistency

AU - Chen, Yun Chun

AU - Huang, Po Hsiang

AU - Yu, Li Yu

AU - Huang, Jia Bin

AU - Yang, Ming Hsuan

AU - Lin, Yen Yu

PY - 2019

Y1 - 2019

N2 - Establishing dense semantic correspondences between object instances remains a challenging problem due to background clutter, significant scale and pose differences, and large intra-class variations. In this paper, we present an end-to-end trainable network for learning semantic correspondences using only matching image pairs without manual keypoint correspondence annotations. To facilitate network training with this weaker form of supervision, we (1) explicitly estimate the foreground regions to suppress the effect of background clutter and (2) develop cycle-consistent losses to enforce the predicted transformations across multiple images to be geometrically plausible and consistent. We train the proposed model using the PF-PASCAL dataset and evaluate the performance on the PF-PASCAL, PF-WILLOW, and TSS datasets. Extensive experimental results show that the proposed approach achieves favorably performance compared to the state-of-the-art. The code and model will be available at https://yunchunchen.github.io/WeakMatchNet/.

AB - Establishing dense semantic correspondences between object instances remains a challenging problem due to background clutter, significant scale and pose differences, and large intra-class variations. In this paper, we present an end-to-end trainable network for learning semantic correspondences using only matching image pairs without manual keypoint correspondence annotations. To facilitate network training with this weaker form of supervision, we (1) explicitly estimate the foreground regions to suppress the effect of background clutter and (2) develop cycle-consistent losses to enforce the predicted transformations across multiple images to be geometrically plausible and consistent. We train the proposed model using the PF-PASCAL dataset and evaluate the performance on the PF-PASCAL, PF-WILLOW, and TSS datasets. Extensive experimental results show that the proposed approach achieves favorably performance compared to the state-of-the-art. The code and model will be available at https://yunchunchen.github.io/WeakMatchNet/.

UR - http://www.scopus.com/inward/record.url?scp=85067235030&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85067235030&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-20893-6_22

DO - 10.1007/978-3-030-20893-6_22

M3 - Conference contribution

AN - SCOPUS:85067235030

SN - 9783030208929

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 347

EP - 362

BT - Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers

A2 - Jawahar, C.V.

A2 - Schindler, Konrad

A2 - Li, Hongdong

A2 - Mori, Greg

PB - Springer Verlag

T2 - 14th Asian Conference on Computer Vision, ACCV 2018

Y2 - 2 December 2018 through 6 December 2018

ER -

Chen YC, Huang PH, Yu LY, Huang JB, Yang MH, Lin YY. Deep Semantic Matching with Foreground Detection and Cycle-Consistency. In Jawahar CV, Schindler K, Li H, Mori G, editors, Computer Vision - ACCV 2018 - 14th Asian Conference on Computer Vision, Revised Selected Papers. Springer Verlag. 2019. p. 347-362. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-20893-6_22

Deep Semantic Matching with Foreground Detection and Cycle-Consistency

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this