Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation
Camgöz, Necati Cihan, Koller, Oscar, Hadfield, Simon and Bowden, Richard (2020) Sign Language Transformers: Joint End-to-end Sign Language Recognition and Translation In: IEEE Conference on Computer Vision and Pattern Recognition 2020, 14th to 19th June, 2020, Seattle, Washington.
|
Text
camgoz2020cvpr.pdf - Accepted version Manuscript Download (849kB) | Preview |
Abstract
Prior work on Sign Language Translation has shown that having a mid-level sign gloss representation(effectively recognizing the individual signs) improves the translation performance drastically. In fact, the current state-of-theart in translation requires gloss level tokenization in order to work. We introduce a novel transformer based architecture that jointly learns Continuous Sign Language Recognition and Translation whilebeing trainable in an end-to-end manner. This is achieved by using a Connectionist Temporal Classification(CTC)loss to bind the recognition and translation problems into a single unified architecture. This joint approach does not require any ground-truth timing information,simultaneously solving two co-dependant sequence to sequence learning problems and leads to significant performance gains. We evaluate the recognition and translation performances of our approaches on the challenging RWTHPHOENIX-Weather-2014T(PHOENIX14T)dataset. Wereport state-of-the-art sign language recognition and translation results achieved by our Sign Language Transformers. Our translation net works out perform both sign video to spoken language and gloss to spoken language translation models, in some cases more than doubling the performance (9.58vs. 21.80BLEU-4Score). We also share new baseline translation results using transformer networks for several other text-to-text sign language translation tasks.
Item Type: | Conference or Workshop Item (Conference Paper) | |||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
Divisions : | Faculty of Engineering and Physical Sciences > Electronic Engineering | |||||||||||||||
Authors : |
|
|||||||||||||||
Date : | 27 February 2020 | |||||||||||||||
Funders : | European Union’s Horizon2020 research and innovation programme | |||||||||||||||
Grant Title : | European Union Horizon 2020 Project | |||||||||||||||
Depositing User : | James Marshall | |||||||||||||||
Date Deposited : | 02 Apr 2020 14:26 | |||||||||||||||
Last Modified : | 02 Apr 2020 14:26 | |||||||||||||||
URI: | http://epubs.surrey.ac.uk/id/eprint/854110 |
Actions (login required)
![]() |
View Item |
Downloads
Downloads per month over past year