SegFlow: Joint Learning for Video Object Segmentation and Optical Flow

Jingchun Cheng; Yi Hsuan Tsai; Shengjin Wang; Ming Hsuan Yang

doi:10.1109/ICCV.2017.81

SegFlow: Joint Learning for Video Object Segmentation and Optical Flow

Jingchun Cheng, Yi Hsuan Tsai, Shengjin Wang, Ming Hsuan Yang

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

317 Citations (Scopus)

Abstract

This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.

Original language	English
Title of host publication	Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	686-695
Number of pages	10
ISBN (Electronic)	9781538610329
DOIs	https://doi.org/10.1109/ICCV.2017.81
Publication status	Published - 2017 Dec 22
Event	16th IEEE International Conference on Computer Vision, ICCV 2017 - Venice, Italy Duration: 2017 Oct 22 → 2017 Oct 29

Publication series

Name	Proceedings of the IEEE International Conference on Computer Vision
Volume	2017-October
ISSN (Print)	1550-5499

Other

Other	16th IEEE International Conference on Computer Vision, ICCV 2017
Country/Territory	Italy
City	Venice
Period	17/10/22 → 17/10/29

Bibliographical note

Funding Information:
This work is supported in part by the NSF CAREER Grant #1149783 and NSF IIS Grant #1152576.

Publisher Copyright:
© 2017 IEEE.

All Science Journal Classification (ASJC) codes

Software
Computer Vision and Pattern Recognition

Access to Document

10.1109/ICCV.2017.81

Cite this

Cheng, J., Tsai, Y. H., Wang, S., & Yang, M. H. (2017). SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. In Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017 (pp. 686-695). [8237343] (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2017-October). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICCV.2017.81

@inproceedings{b59866595c4c40d29e47c9cbb3d2155d,

title = "SegFlow: Joint Learning for Video Object Segmentation and Optical Flow",

abstract = "This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.",

author = "Jingchun Cheng and Tsai, {Yi Hsuan} and Shengjin Wang and Yang, {Ming Hsuan}",

note = "Funding Information: This work is supported in part by the NSF CAREER Grant #1149783 and NSF IIS Grant #1152576. Publisher Copyright: {\textcopyright} 2017 IEEE.; 16th IEEE International Conference on Computer Vision, ICCV 2017 ; Conference date: 22-10-2017 Through 29-10-2017",

year = "2017",

month = dec,

day = "22",

doi = "10.1109/ICCV.2017.81",

language = "English",

series = "Proceedings of the IEEE International Conference on Computer Vision",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "686--695",

booktitle = "Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017",

address = "United States",

}

Cheng, J, Tsai, YH, Wang, S & Yang, MH 2017, SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. in Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017., 8237343, Proceedings of the IEEE International Conference on Computer Vision, vol. 2017-October, Institute of Electrical and Electronics Engineers Inc., pp. 686-695, 16th IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, 17/10/22. https://doi.org/10.1109/ICCV.2017.81

SegFlow: Joint Learning for Video Object Segmentation and Optical Flow. / Cheng, Jingchun; Tsai, Yi Hsuan; Wang, Shengjin et al.
Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 686-695 8237343 (Proceedings of the IEEE International Conference on Computer Vision; Vol. 2017-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - SegFlow

T2 - 16th IEEE International Conference on Computer Vision, ICCV 2017

AU - Cheng, Jingchun

AU - Tsai, Yi Hsuan

AU - Wang, Shengjin

AU - Yang, Ming Hsuan

PY - 2017/12/22

Y1 - 2017/12/22

N2 - This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.

AB - This paper proposes an end-to-end trainable network, SegFlow, for simultaneously predicting pixel-wise object segmentation and optical flow in videos. The proposed SegFlow has two branches where useful information of object segmentation and optical flow is propagated bidirectionally in a unified framework. The segmentation branch is based on a fully convolutional network, which has been proved effective in image segmentation task, and the optical flow branch takes advantage of the FlowNet model. The unified framework is trained iteratively offline to learn a generic notion, and fine-tuned online for specific objects. Extensive experiments on both the video object segmentation and optical flow datasets demonstrate that introducing optical flow improves the performance of segmentation and vice versa, against the state-of-the-art algorithms.

UR - http://www.scopus.com/inward/record.url?scp=85041903685&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85041903685&partnerID=8YFLogxK

U2 - 10.1109/ICCV.2017.81

DO - 10.1109/ICCV.2017.81

M3 - Conference contribution

AN - SCOPUS:85041903685

T3 - Proceedings of the IEEE International Conference on Computer Vision

SP - 686

EP - 695

BT - Proceedings - 2017 IEEE International Conference on Computer Vision, ICCV 2017

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 22 October 2017 through 29 October 2017

ER -

SegFlow: Joint Learning for Video Object Segmentation and Optical Flow

Abstract

Publication series

Other

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this