University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Learning to Sketch with Shortcut Cycle Consistency

Song, J., Pang, K., Song, Yi-Zhe, Xiang, T. and Hospedales, T.M. (2019) Learning to Sketch with Shortcut Cycle Consistency In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 18-22 Jun 2018, Salt Lake City, Utah, USA.

Full text not available from this repository.


To see is to sketch - free-hand sketching naturally builds ties between human and machine vision. In this paper, we present a novel approach for translating an object photo to a sketch, mimicking the human sketching process. This is an extremely challenging task because the photo and sketch domains differ significantly. Furthermore, human sketches exhibit various levels of sophistication and abstraction even when depicting the same object instance in a reference photo. This means that even if photo-sketch pairs are available, they only provide weak supervision signal to learn a translation model. Compared with existing supervised approaches that solve the problem of D(E(photo)) → sketch), where E(·) and D(·) denote encoder and decoder respectively, we take advantage of the inverse problem (e.g., D(E(sketch) → photo), and combine with the unsupervised learning tasks of within-domain reconstruction, all within a multi-task learning framework. Compared with existing unsupervised approaches based on cycle consistency (i.e., D(E(D(E(photo)))) → photo), we introduce a shortcut consistency enforced at the encoder bottleneck (e.g., D(E(photo)) → photo) to exploit the additional self-supervision. Both qualitative and quantitative results show that the proposed model is superior to a number of state-of-the-art alternatives. We also show that the synthetic sketches can be used to train a better fine-grained sketch-based image retrieval (FG-SBIR) model, effectively alleviating the problem of sketch data scarcity.

Item Type: Conference or Workshop Item (Conference Paper)
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Song, J.
Pang, K.
Xiang, T.
Hospedales, T.M.
Date : February 2019
DOI : 10.1109/CVPR.2018.00090
Uncontrolled Keywords : Task analysis; Decoding; Noise measurement; Visualization; Training; Image reconstruction; Unsupervised learning
Related URLs :
Depositing User : Clive Harris
Date Deposited : 03 Jul 2019 11:01
Last Modified : 03 Jul 2019 11:01

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800