University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs

Koller, Oscar, Zargaran, Sepehr, Ney, Hermann and Bowden, Richard (2018) Deep Sign: Enabling Robust Statistical Continuous Sign Language Recognition via Hybrid CNN-HMMs International Journal of Computer Vision, 126 (12). pp. 1311-1325.

Koller2018_Article_DeepSignEnablingRobustStatisti.pdf - Version of Record
Available under License Creative Commons Attribution.

Download (1MB) | Preview
[img] Text - Accepted version Manuscript
Restricted to Repository staff only

Download (1MB)


This manuscript introduces the end-to-end embedding of a CNN into a HMM, while interpreting the outputs of the CNN in a Bayesian framework. The hybrid CNNHMM combines the strong discriminative abilities of CNNs with the sequence modelling capabilities of HMMs. Most current approaches in the field of gesture and sign language recognition disregard the necessity of dealing with sequence data both for training and evaluation. With our presented end-to-end embedding we are able to improve over the state-of-the-art on three challenging benchmark continuous sign language recognition tasks by between 15% and 38% relative reduction in word error rate and up to 20% absolute. We analyse the effect of the CNN structure, network pretraining and number of hidden states. We compare the hybrid modelling to a tandem approach and evaluate the gain of model combination.

Item Type: Article
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Koller, Oscar
Zargaran, Sepehr
Ney, Hermann
Date : 5 October 2018
DOI : 10.1007/s11263-018-1121-3
Copyright Disclaimer : © The Author(s) 2018. This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (, which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.
Uncontrolled Keywords : Sign Language Recognition; Hybrid Approach; CNN-HMM; Statistical Approach; Sequence Modelling
Depositing User : Melanie Hughes
Date Deposited : 05 Sep 2018 13:28
Last Modified : 04 Sep 2019 09:56

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800