Video Object Segmentation Using Kernelized Memory Network With Multiple Kernels

Hongje Seong, Junhyuk Hyun, Euntai Kim

Research output: Contribution to journalArticlepeer-review

2 Citations (Scopus)

Abstract

Semi-supervised video object segmentation (VOS) is to predict the segment of a target object in a video when a ground truth segmentation mask for the target is given in the first frame. Recently, space-time memory networks (STM) have received significant attention as a promising approach for semi-supervised VOS. However, an important point has been overlooked in applying STM to VOS: The solution (=STM) is non-local, but the problem (=VOS) is predominantly local. To solve this mismatch between STM and VOS, we propose new VOS networks called kernelized memory network (KMN) and KMN with multiple kernels (KMN$^{M}$M). Our networks conduct not only Query-to-Memory matching but also Memory-to-Query matching. In Memory-to-Query matching, a kernel is employed to reduce the degree of non-localness of the STM. In addition, we present a Hide-and-Seek strategy in pre-training to handle occlusions effectively. The proposed networks surpass the state-of-the-art results on standard benchmarks by a significant margin (+4% in $\mathcal {J_{M}}$JM on DAVIS 2017 test-dev set). The runtimes of our proposed KMN and KMN$^{M}$M on DAVIS 2016 validation set are 0.12 and 0.13 seconds per frame, respectively, and the two networks have similar computation times to STM.

Original languageEnglish
Pages (from-to)2595-2612
Number of pages18
JournalIEEE transactions on pattern analysis and machine intelligence
Volume45
Issue number2
DOIs
Publication statusPublished - 2023 Feb 1

Bibliographical note

Publisher Copyright:
© 1979-2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics

Fingerprint

Dive into the research topics of 'Video Object Segmentation Using Kernelized Memory Network With Multiple Kernels'. Together they form a unique fingerprint.

Cite this