University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Sign Language Production using Neural Machine Translation and Generative Adversarial Networks

Stoll, Stephanie, Camgöz, Necati Cihan, Hadfield, Simon and Bowden, Richard (2018) Sign Language Production using Neural Machine Translation and Generative Adversarial Networks In: 29th British Machine Vision Conference (BMVC 2018), 03-06 Sep 2018, Northumbria University, Newcastle Upon Tyne, UK.

[img]
Preview
Text
Sign Language Production using Neural Machine Translation and Generative Adversarial Networks.pdf - Accepted version Manuscript

Download (3MB) | Preview

Abstract

We present a novel approach to automatic Sign Language Production using stateof- the-art Neural Machine Translation (NMT) and Image Generation techniques. Our system is capable of producing sign videos from spoken language sentences. Contrary to current approaches that are dependent on heavily annotated data, our approach requires minimal gloss and skeletal level annotations for training. We achieve this by breaking down the task into dedicated sub-processes. We first translate spoken language sentences into sign gloss sequences using an encoder-decoder network. We then find a data driven mapping between glosses and skeletal sequences. We use the resulting pose information to condition a generative model that produces sign language video sequences. We evaluate our approach on the recently released PHOENIX14T Sign Language Translation dataset. We set a baseline for text-to-gloss translation, reporting a BLEU-4 score of 16.34/15.26 on dev/test sets. We further demonstrate the video generation capabilities of our approach by sharing qualitative results of generated sign sequences given their skeletal correspondence.

Item Type: Conference or Workshop Item (Conference Paper)
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
NameEmailORCID
Stoll, Stephanies.m.stoll@surrey.ac.uk
Camgöz, Necati Cihann.camgoz@surrey.ac.uk
Hadfield, Simons.hadfield@surrey.ac.uk
Bowden, RichardR.Bowden@surrey.ac.uk
Date : 3 September 2018
Copyright Disclaimer : © 2018. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.
Related URLs :
Depositing User : Clive Harris
Date Deposited : 31 Jul 2018 10:10
Last Modified : 03 Sep 2018 02:08
URI: http://epubs.surrey.ac.uk/id/eprint/848809

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year


Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800