University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Reverberant Speech Separation Based on Audio-visual Dictionary Learning and Binaural Cues

Liu, Q, Wang, W, Jackson, PJB and Barnard, M (2012) Reverberant Speech Separation Based on Audio-visual Dictionary Learning and Binaural Cues In: IEEE Statistical Signal Processing Workshop (SSP), 2012-08-05 - 2012-08-08, Ann Abor, USA.

Available under License : See the attached licence file.

Download (320kB)
Text (licence)

Download (33kB)


Probabilistic models of binaural cues, such as the interaural phase difference (IPD) and the interaural level difference (ILD), can be used to obtain the audio mask in the time-frequency (TF) domain, for source separation of binaural mixtures. Those models are, however, often degraded by acoustic noise. In contrast, the video stream contains relevant information about the synchronous audio stream that is not affected by acoustic noise. In this paper, we present a novel method for modeling the audio-visual (AV) coherence based on dictionary learning. A visual mask is constructed from the video signal based on the learnt AV dictionary, and incorporated with the audio mask to obtain a noise-robust audio-visual mask, which is then applied to the binaural signal for source separation. We tested our algorithm on the XM2VTS database, and observed considerable performance improvement for noise corrupted signals.

Item Type: Conference or Workshop Item (Conference Paper)
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering > Centre for Vision Speech and Signal Processing
Authors :
Liu, Q
Wang, W
Jackson, PJB
Barnard, M
Date : 5 August 2012
DOI : 10.1109/SSP.2012.6319789
Contributors :
Additional Information : © 2012IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Depositing User : Symplectic Elements
Date Deposited : 17 May 2013 17:15
Last Modified : 31 Oct 2017 15:04

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800