University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Adversarial Training for Multi-Channel Sign Language Production

Saunders, Ben, Camgöz, Necati Cihan and Bowden, Richard (2020) Adversarial Training for Multi-Channel Sign Language Production The 31st British Machine Vision Virtual Conference.

BMVC20_Adversarial_MutilChannel_SLP.pdf - Accepted version Manuscript

Download (7MB) | Preview


Sign Languages are rich multi-channel languages, requiring articulation of both manual (hands) and non-manual (face and body) features in a precise, intricate manner. Sign Language Production (SLP), the automatic translation from spoken to sign languages, must embody this full sign morphology to be truly understandable by the Deaf community. Previous work has mainly focused on manual feature production, with an under-articulated output caused by regression to the mean. In this paper, we propose an Adversarial Multi-Channel approach to SLP. We frame sign production as a minimax game between a transformer-based Generator and a conditional Discriminator. Our adversarial discriminator evaluates the realism of sign production conditioned on the source text, pushing the generator towards a realistic and articulate output. Additionally, we fully encapsulate sign articulators with the inclusion of non-manual features, producing facial features and mouthing patterns. We evaluate on the challenging RWTH-PHOENIX-Weather-2014T (PHOENIX14T) dataset, and report state-of-the art SLP back-translation performance for manual production. We set new benchmarks for the production of multi-channel sign to underpin future research into realistic SLP.

Item Type: Article
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Camgöz, Necati
Date : 5 August 2020
Grant Title : SNSF Sinergia project 'SMILE'
Copyright Disclaimer : c 2020. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.
Projects : SNSF Sinergia project 'SMILE', EPSRC project ‘ExTOL’
Additional Information : Embargo OK Metadata Pending Awaiting final version published online.
Depositing User : James Marshall
Date Deposited : 17 Aug 2020 10:27
Last Modified : 07 Sep 2020 02:08

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800