University of Surrey

Test tubes in the lab Research in the ATI Dance Research

SignSynth: Data-Driven Sign Language Video Generation

Stoll, Stephanie, Hadfield, Simon and Bowden, Richard (2020) SignSynth: Data-Driven Sign Language Video Generation In: Eighth International Workshop on Assistive Computer Vision and Robotics, 23-28 Aug 2020, Virtual Conference.

W06P18.pdf - Accepted version Manuscript

Download (2MB) | Preview


We present SignSynth, a fully automatic and holistic approach to generating sign language video. Traditionally, Sign Language Production (SLP) relies on animating 3D avatars using expensively annotated data, but so far this approach has not been able to simultaneously provide a realistic, and scalable solution. We introduce a gloss2pose network architecture that is capable of generating human pose sequences conditioned on glosses.1 Combined with a generative adversarial pose2video network, we are able to produce natural-looking, high definition sign language video. For sign pose sequence generation, we outperform the SotA by a factor of 18, with a Mean Square Error of 1.0673 in pixels. For video generation we report superior results on three broadcast quality assessment metrics. To evaluate our full gloss-to-video pipeline we introduce two novel error metrics, to assess the perceptual quality and sign representativeness of generated videos. We present promising results, significantly outperforming the SotA in both metrics. Finally we evaluate our approach qualitatively by analysing example sequences.

Item Type: Conference or Workshop Item (Conference Paper)
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Date : 3 August 2020
Funders : SNSF Sinergia, European Union's Horizon 2020, EPSRC
Grant Title : SNSF Sinergia
Projects : SNSF Sinergia project 'SMILE', EPSRC project 'ExTOL'
Uncontrolled Keywords : Sign Language; Pose Generation; Human Motion
Additional Information : Embargo OK Metadata Pending Awaiting final version published online
Depositing User : James Marshall
Date Deposited : 26 Aug 2020 10:10
Last Modified : 26 Aug 2020 10:14

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800