Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization

Jungin Park; Jiyoung Lee; Sangryul Jeon; Seungryong Kim; Kwanghoon Sohn

doi:10.1109/ICIP.2019.8803589

Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization

Jungin Park, Jiyoung Lee, Sangryul Jeon, Seungryong Kim, Kwanghoon Sohn

Department of Electrical and Electronic Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

4 Citations (Scopus)

Abstract

This paper presents a novel deep architecture for weakly-supervised temporal action localization that not only generates segment-level action responses but also propagates segment-level responses to the neighborhood in a form of graph Laplacian regularization. Specifically, our approach consists of two sub-modules; a class activation module to estimate the action score map over time through the action classifiers, and a graph regularization module to refine the estimated action score map by solving a quadratic programming problem with the predicted segment-level semantic affinities. Since these two modules are integrated with fully differentiable layers, the proposed networks can be jointly trained in an end-to-end manner. Experimental results on Thumos14 and ActivityNet1.2 demonstrate that the proposed method provides outstanding performances in weakly-supervised temporal action localization.

Original language	English
Title of host publication	2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings
Publisher	IEEE Computer Society
Pages	3701-3705
Number of pages	5
ISBN (Electronic)	9781538662496
DOIs	https://doi.org/10.1109/ICIP.2019.8803589
Publication status	Published - 2019 Sept
Event	26th IEEE International Conference on Image Processing, ICIP 2019 - Taipei, Taiwan, Province of China Duration: 2019 Sept 22 → 2019 Sept 25

Publication series

Name	Proceedings - International Conference on Image Processing, ICIP
Volume	2019-September
ISSN (Print)	1522-4880

Conference

Conference	26th IEEE International Conference on Image Processing, ICIP 2019
Country/Territory	Taiwan, Province of China
City	Taipei
Period	19/9/22 → 19/9/25

Bibliographical note

Funding Information:
This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.2017K1A3A1A16066838). (Corresponding author:Kwanghoon Sohn.)

Publisher Copyright:
© 2019 IEEE.

All Science Journal Classification (ASJC) codes

Software
Computer Vision and Pattern Recognition
Signal Processing

Access to Document

10.1109/ICIP.2019.8803589

Cite this

Park, J., Lee, J., Jeon, S., Kim, S., & Sohn, K. (2019). Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization. In 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings (pp. 3701-3705). [8803589] (Proceedings - International Conference on Image Processing, ICIP; Vol. 2019-September). IEEE Computer Society. https://doi.org/10.1109/ICIP.2019.8803589

@inproceedings{44b421ef57424747ada27759834cfc7d,

title = "Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization",

abstract = "This paper presents a novel deep architecture for weakly-supervised temporal action localization that not only generates segment-level action responses but also propagates segment-level responses to the neighborhood in a form of graph Laplacian regularization. Specifically, our approach consists of two sub-modules; a class activation module to estimate the action score map over time through the action classifiers, and a graph regularization module to refine the estimated action score map by solving a quadratic programming problem with the predicted segment-level semantic affinities. Since these two modules are integrated with fully differentiable layers, the proposed networks can be jointly trained in an end-to-end manner. Experimental results on Thumos14 and ActivityNet1.2 demonstrate that the proposed method provides outstanding performances in weakly-supervised temporal action localization.",

author = "Jungin Park and Jiyoung Lee and Sangryul Jeon and Seungryong Kim and Kwanghoon Sohn",

note = "Funding Information: This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.2017K1A3A1A16066838). (Corresponding author:Kwanghoon Sohn.) Publisher Copyright: {\textcopyright} 2019 IEEE.; 26th IEEE International Conference on Image Processing, ICIP 2019 ; Conference date: 22-09-2019 Through 25-09-2019",

year = "2019",

month = sep,

doi = "10.1109/ICIP.2019.8803589",

language = "English",

series = "Proceedings - International Conference on Image Processing, ICIP",

publisher = "IEEE Computer Society",

pages = "3701--3705",

booktitle = "2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings",

address = "United States",

}

Park, J, Lee, J, Jeon, S, Kim, S & Sohn, K 2019, Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization. in 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings., 8803589, Proceedings - International Conference on Image Processing, ICIP, vol. 2019-September, IEEE Computer Society, pp. 3701-3705, 26th IEEE International Conference on Image Processing, ICIP 2019, Taipei, Taiwan, Province of China, 19/9/22. https://doi.org/10.1109/ICIP.2019.8803589

Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization. / Park, Jungin; Lee, Jiyoung; Jeon, Sangryul et al.
2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings. IEEE Computer Society, 2019. p. 3701-3705 8803589 (Proceedings - International Conference on Image Processing, ICIP; Vol. 2019-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization

AU - Park, Jungin

AU - Lee, Jiyoung

AU - Jeon, Sangryul

AU - Kim, Seungryong

AU - Sohn, Kwanghoon

N1 - Funding Information: This work was supported by the National Research Foundation of Korea(NRF) grant funded by the Korea government(MSIT) (No.2017K1A3A1A16066838). (Corresponding author:Kwanghoon Sohn.) Publisher Copyright: © 2019 IEEE.

PY - 2019/9

Y1 - 2019/9

N2 - This paper presents a novel deep architecture for weakly-supervised temporal action localization that not only generates segment-level action responses but also propagates segment-level responses to the neighborhood in a form of graph Laplacian regularization. Specifically, our approach consists of two sub-modules; a class activation module to estimate the action score map over time through the action classifiers, and a graph regularization module to refine the estimated action score map by solving a quadratic programming problem with the predicted segment-level semantic affinities. Since these two modules are integrated with fully differentiable layers, the proposed networks can be jointly trained in an end-to-end manner. Experimental results on Thumos14 and ActivityNet1.2 demonstrate that the proposed method provides outstanding performances in weakly-supervised temporal action localization.

AB - This paper presents a novel deep architecture for weakly-supervised temporal action localization that not only generates segment-level action responses but also propagates segment-level responses to the neighborhood in a form of graph Laplacian regularization. Specifically, our approach consists of two sub-modules; a class activation module to estimate the action score map over time through the action classifiers, and a graph regularization module to refine the estimated action score map by solving a quadratic programming problem with the predicted segment-level semantic affinities. Since these two modules are integrated with fully differentiable layers, the proposed networks can be jointly trained in an end-to-end manner. Experimental results on Thumos14 and ActivityNet1.2 demonstrate that the proposed method provides outstanding performances in weakly-supervised temporal action localization.

UR - http://www.scopus.com/inward/record.url?scp=85076823912&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85076823912&partnerID=8YFLogxK

U2 - 10.1109/ICIP.2019.8803589

DO - 10.1109/ICIP.2019.8803589

M3 - Conference contribution

AN - SCOPUS:85076823912

T3 - Proceedings - International Conference on Image Processing, ICIP

SP - 3701

EP - 3705

BT - 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings

PB - IEEE Computer Society

T2 - 26th IEEE International Conference on Image Processing, ICIP 2019

Y2 - 22 September 2019 through 25 September 2019

ER -

Park J, Lee J, Jeon S, Kim S, Sohn K. Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization. In 2019 IEEE International Conference on Image Processing, ICIP 2019 - Proceedings. IEEE Computer Society. 2019. p. 3701-3705. 8803589. (Proceedings - International Conference on Image Processing, ICIP). doi: 10.1109/ICIP.2019.8803589

Graph Regularization Network with Semantic Affinity for Weakly-Supervised Temporal Action Localization

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this