FaceFilter: Audio-visual speech separation using still images

Soo Whan Chung, Soyeon Choe, Joon Son Chung, Hong Goo Kang

Research output: Contribution to journalConference articlepeer-review

32 Citations (Scopus)

Abstract

The objective of this paper is to separate a target speaker's speech from a mixture of two speakers using a deep audiovisual speech separation network. Unlike previous works that used lip movement on video clips or pre-enrolled speaker information as an auxiliary conditional feature, we use a single face image of the target speaker. In this task, the conditional feature is obtained from facial appearance in cross-modal biometric task, where audio and visual identity representations are shared in latent space. Learnt identities from facial images enforce the network to isolate matched speakers and extract the voices from mixed speech. It solves the permutation problem caused by swapped channel outputs, frequently occurred in speech separation tasks. The proposed method is far more practical than video-based speech separation since user profile images are readily available on many platforms. Also, unlike speaker-aware separation methods, it is applicable on separation with unseen speakers who have never been enrolled before. We show strong qualitative and quantitative results on challenging real-world examples.

Original languageEnglish
Pages (from-to)3481-3485
Number of pages5
JournalProceedings of the Annual Conference of the International Speech Communication Association, INTERSPEECH
Volume2020-October
DOIs
Publication statusPublished - 2020
Event21st Annual Conference of the International Speech Communication Association, INTERSPEECH 2020 - Shanghai, China
Duration: 2020 Oct 252020 Oct 29

Bibliographical note

Publisher Copyright:
Copyright © 2020 ISCA

All Science Journal Classification (ASJC) codes

  • Language and Linguistics
  • Human-Computer Interaction
  • Signal Processing
  • Software
  • Modelling and Simulation

Fingerprint

Dive into the research topics of 'FaceFilter: Audio-visual speech separation using still images'. Together they form a unique fingerprint.

Cite this