Multi-Task Structure-Aware Context Modeling for Robust Keypoint-Based Object Tracking

Xi Li, Liming Zhao, Wei Ji, Yiming Wu, Fei Wu, Ming Hsuan Yang, Dacheng Tao, Ian Reid

Research output: Contribution to journalArticlepeer-review

14 Citations (Scopus)


In the fields of computer vision and graphics, keypoint-based object tracking is a fundamental and challenging problem, which is typically formulated in a spatio-temporal context modeling framework. However, many existing keypoint trackers are incapable of effectively modeling and balancing the following three aspects in a simultaneous manner: temporal model coherence across frames, spatial model consistency within frames, and discriminative feature construction. To address this problem, we propose a robust keypoint tracker based on spatio-temporal multi-task structured output optimization driven by discriminative metric learning. Consequently, temporal model coherence is characterized by multi-task structured keypoint model learning over several adjacent frames; spatial model consistency is modeled by solving a geometric verification based structured learning problem; discriminative feature construction is enabled by metric learning to ensure the intra-class compactness and inter-class separability. To achieve the goal of effective object tracking, we jointly optimize the above three modules in a spatio-temporal multi-task learning scheme. Furthermore, we incorporate this joint learning scheme into both single-object and multi-object tracking scenarios, resulting in robust tracking results. Experiments over several challenging datasets have justified the effectiveness of our single-object and multi-object trackers against the state-of-the-art.

Original languageEnglish
Article number8322285
Pages (from-to)915-927
Number of pages13
JournalIEEE transactions on pattern analysis and machine intelligence
Issue number4
Publication statusPublished - 2019 Apr 1

Bibliographical note

Funding Information:
We greatly appreciate Yueting Zhuang and Jun Xiao for their valuable comments and suggestions on this work. This work was supported in part by the National Natural Science Foundation of China under Grants (U1509206, 61472353, and 61751209), in part by the National Basic Research Program of China under Grant Grant 2015CB352302, and partially funded by the MOE-Microsoft Key Laboratory of Visual Perception, Zhejiang University.

Publisher Copyright:
© 1979-2012 IEEE.

All Science Journal Classification (ASJC) codes

  • Software
  • Computer Vision and Pattern Recognition
  • Computational Theory and Mathematics
  • Artificial Intelligence
  • Applied Mathematics


Dive into the research topics of 'Multi-Task Structure-Aware Context Modeling for Robust Keypoint-Based Object Tracking'. Together they form a unique fingerprint.

Cite this