University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Progressive Transformers for End-to-End Sign Language Production

Saunders, Ben, Camgöz, Necati Cihan and Bowden, Richard (2020) Progressive Transformers for End-to-End Sign Language Production In: European Conference on Computer Vision (ECCV), 23-28 Aug 2020, Virtual Conference.

Progressive_Transformers_for_E2E_Sign_Language_Production_CRC.pdf - Accepted version Manuscript

Download (6MB) | Preview


The goal of automatic Sign Language Production (SLP) is to translate spoken language to a continuous stream of sign language video at a level comparable to a human translator. If this was achievable, then it would revolutionise Deaf hearing communications. Previous work on predominantly isolated SLP has shown the need for architectures that are better suited to the continuous domain of full sign sequences. In this paper, we propose Progressive Transformers, the first SLP model to translate from discrete spoken language sentences to continuous 3D sign pose sequences in an end-to-end manner. A novel counter decoding technique is introduced, that enables continuous sequence generation at training and inference. We present two model configurations, an end-to end network that produces sign direct from text and a stacked network that utilises a gloss intermediary. We also provide several data augmentation processes to overcome the problem of drift and drastically improve the performance of SLP models. We propose a back translation evaluation mechanism for SLP, presenting benchmark quantitative results on the challenging RWTH-PHOENIXWeather- 2014T (PHOENIX14T) dataset and setting baselines for future research. Code available at ProgressiveTransformersSLP.

Item Type: Conference or Workshop Item (Conference Paper)
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Camgöz, Necati
Date : 2 July 2020
Funders : SNSF Sinergia project, Horizon 2020, EPSRC, NVIDIA Corporation
Grant Title : SNSF Sinergia project 'SMILE'
Projects : SNSF Sinergia project 'SMILE'
Uncontrolled Keywords : Sign Language Production, Continuous Sequence Synthesis, Transformers, Sequence-to-Sequence, Human Pose Generation
Additional Information : Embargo OK Metadata Pending
Depositing User : James Marshall
Date Deposited : 20 Jul 2020 08:58
Last Modified : 23 Aug 2020 02:08

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800