2.5D visual relationship detection

Yu Chuan Su, Soravit Changpinyo, Xiangning Chen, Sathish Thoppay, Cho Jui Hsieh, Lior Shapira, Radu Soricut, Hartwig Adam, Matthew Brown, Ming Hsuan Yang, Boqing Gong

Research output: Contribution to journalArticlepeer-review


Visual 2.5D perception involves understanding the semantics and geometry of a scene through reasoning about object relationships with respect to the viewer. However, existing works in visual recognition primarily focus on the semantics. To bridge this gap, we study 2.5D visual relationship detection (2.5VRD), in which the goal is to jointly detect objects and predict their relative depth and occlusion relationships. Unlike general VRD, 2.5VRD is egocentric, using the camera's viewpoint as a common reference for all 2.5D relationships. Unlike depth estimation, 2.5VRD is object-centric and does not only focus on depth. To enable progress on this task, we construct a new dataset consisting of 220K human-annotated 2.5D relationships among 512K objects from 11K images. We analyze this dataset and conduct extensive experiments including benchmarking multiple state-of-the-art VRD models on this task. Experimental results show that existing models largely rely on semantic cues and simple heuristics to solve 2.5VRD, motivating further research on models for 2.5D perception. We will make our dataset and source code publicly available.

Original languageEnglish
Article number103557
JournalComputer Vision and Image Understanding
Publication statusPublished - 2022 Nov

Bibliographical note

Publisher Copyright:
© 2022 Elsevier Inc.

All Science Journal Classification (ASJC) codes

  • Software
  • Signal Processing
  • Computer Vision and Pattern Recognition


Dive into the research topics of '2.5D visual relationship detection'. Together they form a unique fingerprint.

Cite this