Retrieving and classifying affective images via deep metric learning

Jufeng Yang, Dongyu She, Yu Kun Lai, Ming Hsuan Yang

Research output: Chapter in Book/Report/Conference proceedingConference contribution

52 Citations (Scopus)


Affective image understanding has been extensively studied in the last decade since more and more users express emotion via visual contents. While current algorithms based on convolutional neural networks aim to distinguish emotional categories in a discrete label space, the task is inherently ambiguous. This is mainly because emotional labels with the same polarity (i.e., positive or negative) are highly related, which is different from concrete object concepts such as cat, dog and bird. To the best of our knowledge, few methods focus on leveraging such characteristic of emotions for affective image understanding. In this work, we address the problem of understanding affective images via deep metric learning and propose a multi-task deep framework to optimize both retrieval and classification goals. We propose the sentiment constraints adapted from the triplet constraints, which are able to explore the hierarchical relation of emotion labels. We further exploit the sentiment vector as an effective representation to distinguish affective images utilizing the texture representation derived from convolutional layers. Extensive evaluations on four widely-used affective datasets, i.e., Flickr and Instagram, IAPSa, Art Photo, and Abstract Paintings, demonstrate that the proposed algorithm performs favorably against the state-of-the-art methods on both affective image retrieval and classification tasks.

Original languageEnglish
Title of host publication32nd AAAI Conference on Artificial Intelligence, AAAI 2018
PublisherAAAI press
Number of pages8
ISBN (Electronic)9781577358008
Publication statusPublished - 2018
Event32nd AAAI Conference on Artificial Intelligence, AAAI 2018 - New Orleans, United States
Duration: 2018 Feb 22018 Feb 7

Publication series

Name32nd AAAI Conference on Artificial Intelligence, AAAI 2018


Other32nd AAAI Conference on Artificial Intelligence, AAAI 2018
Country/TerritoryUnited States
CityNew Orleans

Bibliographical note

Funding Information:
This research was sponsored by NSFC (61620106008, 61572264), CAST (YESS20150117), Huawei Innovation Research Program (HIRP), and IBM Global SUR award.

Publisher Copyright:
Copyright © 2018, Association for the Advancement of Artificial Intelligence ( All rights reserved.

All Science Journal Classification (ASJC) codes

  • Artificial Intelligence


Dive into the research topics of 'Retrieving and classifying affective images via deep metric learning'. Together they form a unique fingerprint.

Cite this