Reverberant Speech Separation Based on Audio-visual Dictionary Learning and Binaural Cues
Liu, Q, Wang, W, Jackson, PJB and Barnard, M (2012) Reverberant Speech Separation Based on Audio-visual Dictionary Learning and Binaural Cues In: IEEE Statistical Signal Processing Workshop (SSP), 2012-08-05 - 2012-08-08, Ann Abor, USA.
Available under License : See the attached licence file.
Probabilistic models of binaural cues, such as the interaural phase difference (IPD) and the interaural level difference (ILD), can be used to obtain the audio mask in the time-frequency (TF) domain, for source separation of binaural mixtures. Those models are, however, often degraded by acoustic noise. In contrast, the video stream contains relevant information about the synchronous audio stream that is not affected by acoustic noise. In this paper, we present a novel method for modeling the audio-visual (AV) coherence based on dictionary learning. A visual mask is constructed from the video signal based on the learnt AV dictionary, and incorporated with the audio mask to obtain a noise-robust audio-visual mask, which is then applied to the binaural signal for source separation. We tested our algorithm on the XM2VTS database, and observed considerable performance improvement for noise corrupted signals.
|Item Type:||Conference or Workshop Item (Conference Paper)|
|Divisions :||Faculty of Engineering and Physical Sciences > Electronic Engineering > Centre for Vision Speech and Signal Processing|
|Date :||5 August 2012|
|Identification Number :||https://doi.org/10.1109/SSP.2012.6319789|
|Additional Information :||© 2012IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.|
|Depositing User :||Symplectic Elements|
|Date Deposited :||17 May 2013 17:15|
|Last Modified :||23 Sep 2013 20:07|
Actions (login required)
Downloads per month over past year