Learning to insert an object instance into an image in a semantically coherent manner is a challenging and interesting problem. Solving it requires (a) determining a location to place an object in the scene and (b) determining its appearance at the location. Such an object insertion model can potentially facilitate numerous image editing and scene parsing applications. In this paper, we propose an end-to-end trainable neural network for the task of inserting an object instance mask of a specified class into the semantic label map of an image. Our network consists of two generative modules where one determines where the inserted object mask should be (i.e., location and scale) and the other determines what the object mask shape (and pose) should look like. The two modules are connected together via a spatial transformation network and jointly trained. We devise a learning procedure that leverage both supervised and unsupervised data and show our model can insert an object at diverse locations with various appearances. We conduct extensive experimental validations with comparisons to strong baselines to verify the effectiveness of the proposed network. Code is available at https://github.com/NVlabs/Instance_Insertion.
|Number of pages||11|
|Journal||Advances in Neural Information Processing Systems|
|Publication status||Published - 2018|
|Event||32nd Conference on Neural Information Processing Systems, NeurIPS 2018 - Montreal, Canada|
Duration: 2018 Dec 2 → 2018 Dec 8
Bibliographical noteFunding Information:
Acknowledgements This work was conducted in NVIDIA. Ming-Hsuan Yang is supported in part by the NSF CAREER Grant #1149783 and gifts from NVIDIA.
This work was conducted in NVIDIA. Ming-Hsuan Yang is supported in part by the NSF CAREER Grant #1149783 and gifts from NVIDIA.
© 2018 Curran Associates Inc.All rights reserved.
All Science Journal Classification (ASJC) codes
- Computer Networks and Communications
- Information Systems
- Signal Processing