A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences

Jaeil Park; Seok Jun Bu; Sung Bae Cho

doi:10.1007/978-3-031-15471-3_38

A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences

Jaeil Park, Seok Jun Bu, Sung Bae Cho

College of Computing

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

With the rapid increase in the amount of video data, efficient object recognition is mandatory for a system capable of automatically performing question and answering. In particular, real-world video environments with numerous types of objects and complex relationships require extensive knowledge representation and inference algorithms with the properties and relations of objects. In this paper, we propose a hybrid neuro-symbolic AI system that handles scene-graph of real-world video data. The method combines neural networks that generate scene graphs in consideration of the relationship between objects on real roads and symbol-based inference algorithms for responding to questions. We define object properties, relationships, and question coverage to cover the real-world objects in pedestrian video and traverse a scene-graph to perform complex visual question-answering. We have demonstrated the superiority of the proposed method by confirming that it answered with 99.71% accuracy to 5-types of questions in a pedestrian video environment.

Original language	English
Title of host publication	Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings
Editors	Pablo García Bringas, Hilde Pérez García, Francisco Javier Martínez de Pisón, José Ramón Villar Flecha, Alicia Troncoso Lora, Enrique A. de la Cal, Alvaro Herrero, Francisco Martínez Álvarez, Giuseppe Psaila, Hector Quintián, Emilio Corchado
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	443-454
Number of pages	12
ISBN (Print)	9783031154706
DOIs	https://doi.org/10.1007/978-3-031-15471-3_38
Publication status	Published - 2022
Event	17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022 - Salamancaa, Spain Duration: 2022 Sept 5 → 2022 Sept 7

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13469 LNAI
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022
Country/Territory	Spain
City	Salamancaa
Period	22/9/5 → 22/9/7

Bibliographical note

Publisher Copyright:
© 2022, Springer Nature Switzerland AG.

All Science Journal Classification (ASJC) codes

Theoretical Computer Science
Computer Science(all)

Access to Document

10.1007/978-3-031-15471-3_38

Cite this

Park, J., Bu, S. J., & Cho, S. B. (2022). A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences. In P. García Bringas, H. Pérez García, F. J. Martínez de Pisón, J. R. Villar Flecha, A. Troncoso Lora, E. A. de la Cal, A. Herrero, F. Martínez Álvarez, G. Psaila, H. Quintián, & E. Corchado (Eds.), Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings (pp. 443-454). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13469 LNAI). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-15471-3_38

Park, Jaeil ; Bu, Seok Jun ; Cho, Sung Bae. / A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences. Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings. editor / Pablo García Bringas ; Hilde Pérez García ; Francisco Javier Martínez de Pisón ; José Ramón Villar Flecha ; Alicia Troncoso Lora ; Enrique A. de la Cal ; Alvaro Herrero ; Francisco Martínez Álvarez ; Giuseppe Psaila ; Hector Quintián ; Emilio Corchado. Springer Science and Business Media Deutschland GmbH, 2022. pp. 443-454 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{7fda841ef5164cf4a5f45fcb25792d4f,

title = "A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences",

abstract = "With the rapid increase in the amount of video data, efficient object recognition is mandatory for a system capable of automatically performing question and answering. In particular, real-world video environments with numerous types of objects and complex relationships require extensive knowledge representation and inference algorithms with the properties and relations of objects. In this paper, we propose a hybrid neuro-symbolic AI system that handles scene-graph of real-world video data. The method combines neural networks that generate scene graphs in consideration of the relationship between objects on real roads and symbol-based inference algorithms for responding to questions. We define object properties, relationships, and question coverage to cover the real-world objects in pedestrian video and traverse a scene-graph to perform complex visual question-answering. We have demonstrated the superiority of the proposed method by confirming that it answered with 99.71% accuracy to 5-types of questions in a pedestrian video environment.",

author = "Jaeil Park and Bu, {Seok Jun} and Cho, {Sung Bae}",

note = "Publisher Copyright: {\textcopyright} 2022, Springer Nature Switzerland AG.; 17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022 ; Conference date: 05-09-2022 Through 07-09-2022",

year = "2022",

doi = "10.1007/978-3-031-15471-3_38",

language = "English",

isbn = "9783031154706",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "443--454",

editor = "{Garc{\'i}a Bringas}, Pablo and {P{\'e}rez Garc{\'i}a}, Hilde and {Mart{\'i}nez de Pis{\'o}n}, {Francisco Javier} and {Villar Flecha}, {Jos{\'e} Ram{\'o}n} and {Troncoso Lora}, Alicia and {de la Cal}, {Enrique A.} and Alvaro Herrero and {Mart{\'i}nez {\'A}lvarez}, Francisco and Giuseppe Psaila and Hector Quinti{\'a}n and Emilio Corchado",

booktitle = "Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings",

address = "Germany",

}

Park, J, Bu, SJ & Cho, SB 2022, A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences. in P García Bringas, H Pérez García, FJ Martínez de Pisón, JR Villar Flecha, A Troncoso Lora, EA de la Cal, A Herrero, F Martínez Álvarez, G Psaila, H Quintián & E Corchado (eds), Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13469 LNAI, Springer Science and Business Media Deutschland GmbH, pp. 443-454, 17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022, Salamancaa, Spain, 22/9/5. https://doi.org/10.1007/978-3-031-15471-3_38

A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences. / Park, Jaeil; Bu, Seok Jun; Cho, Sung Bae.
Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings. ed. / Pablo García Bringas; Hilde Pérez García; Francisco Javier Martínez de Pisón; José Ramón Villar Flecha; Alicia Troncoso Lora; Enrique A. de la Cal; Alvaro Herrero; Francisco Martínez Álvarez; Giuseppe Psaila; Hector Quintián; Emilio Corchado. Springer Science and Business Media Deutschland GmbH, 2022. p. 443-454 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13469 LNAI).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences

AU - Park, Jaeil

AU - Bu, Seok Jun

AU - Cho, Sung Bae

PY - 2022

Y1 - 2022

N2 - With the rapid increase in the amount of video data, efficient object recognition is mandatory for a system capable of automatically performing question and answering. In particular, real-world video environments with numerous types of objects and complex relationships require extensive knowledge representation and inference algorithms with the properties and relations of objects. In this paper, we propose a hybrid neuro-symbolic AI system that handles scene-graph of real-world video data. The method combines neural networks that generate scene graphs in consideration of the relationship between objects on real roads and symbol-based inference algorithms for responding to questions. We define object properties, relationships, and question coverage to cover the real-world objects in pedestrian video and traverse a scene-graph to perform complex visual question-answering. We have demonstrated the superiority of the proposed method by confirming that it answered with 99.71% accuracy to 5-types of questions in a pedestrian video environment.

AB - With the rapid increase in the amount of video data, efficient object recognition is mandatory for a system capable of automatically performing question and answering. In particular, real-world video environments with numerous types of objects and complex relationships require extensive knowledge representation and inference algorithms with the properties and relations of objects. In this paper, we propose a hybrid neuro-symbolic AI system that handles scene-graph of real-world video data. The method combines neural networks that generate scene graphs in consideration of the relationship between objects on real roads and symbol-based inference algorithms for responding to questions. We define object properties, relationships, and question coverage to cover the real-world objects in pedestrian video and traverse a scene-graph to perform complex visual question-answering. We have demonstrated the superiority of the proposed method by confirming that it answered with 99.71% accuracy to 5-types of questions in a pedestrian video environment.

UR - http://www.scopus.com/inward/record.url?scp=85139002994&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85139002994&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-15471-3_38

DO - 10.1007/978-3-031-15471-3_38

M3 - Conference contribution

AN - SCOPUS:85139002994

SN - 9783031154706

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 443

EP - 454

BT - Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings

A2 - García Bringas, Pablo

A2 - Pérez García, Hilde

A2 - Martínez de Pisón, Francisco Javier

A2 - Villar Flecha, José Ramón

A2 - Troncoso Lora, Alicia

A2 - de la Cal, Enrique A.

A2 - Herrero, Alvaro

A2 - Martínez Álvarez, Francisco

A2 - Psaila, Giuseppe

A2 - Quintián, Hector

A2 - Corchado, Emilio

PB - Springer Science and Business Media Deutschland GmbH

T2 - 17th International Conference on Hybrid Artificial Intelligence Systems, HAIS 2022

Y2 - 5 September 2022 through 7 September 2022

ER -

Park J, Bu SJ, Cho SB. A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences. In García Bringas P, Pérez García H, Martínez de Pisón FJ, Villar Flecha JR, Troncoso Lora A, de la Cal EA, Herrero A, Martínez Álvarez F, Psaila G, Quintián H, Corchado E, editors, Hybrid Artificial Intelligent Systems - 17th International Conference, HAIS 2022, Proceedings. Springer Science and Business Media Deutschland GmbH. 2022. p. 443-454. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-15471-3_38

A Neuro-Symbolic AI System for Visual Question Answering in Pedestrian Video Sequences

Abstract

Publication series

Conference

Bibliographical note

All Science Journal Classification (ASJC) codes

Access to Document

Other files and links

Fingerprint

Cite this