University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Fine-Grained Instance-Level Sketch-Based Image Retrieval

Yu, Qian, Song, Jifei, Song, Yi-Zhe, Xiang, Tao and Hospedales, Timothy M. (2020) Fine-Grained Instance-Level Sketch-Based Image Retrieval International Journal of Computer Vision.

Full text not available from this repository.


The problem of fine-grained sketch-based image retrieval (FG-SBIR) is defined and investigated in this paper. In FG-SBIR, free-hand human sketch images are used as queries to retrieve photo images containing the same object instances. It is thus a cross-domain (sketch to photo) instance-level retrieval task. It is an extremely challenging problem because (i) visual comparisons and matching need to be executed under large domain gap, i.e., from black and white line drawing sketches to colour photos; (ii) it requires to capture the fine-grained (dis)similarities of sketches and photo images while free-hand sketches drawn by different people present different levels of deformation and expressive interpretation; and (iii) annotated cross-domain fine-grained SBIR datasets are scarce, challenging many state-of-the-art machine learning techniques, particularly those based on deep learning. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based object instance retrieval application. Specifically, a new large-scale FG-SBIR database is introduced which is carefully designed to reflect the real-world application scenarios. A deep cross-domain matching model is then formulated to solve the intrinsic drawing style variability, large domain gap issues, and capture instance-level discriminative features. It distinguishes itself by a carefully designed attention module. Extensive experiments on the new dataset demonstrate the effectiveness of the proposed model and validate the need for a rigorous definition of the FG-SBIR problem and collecting suitable datasets.

Item Type: Article
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Yu, Qian
Song, Jifei
Hospedales, Timothy M.
Date : 30 September 2020
DOI : 10.1007/s11263-020-01382-3
Copyright Disclaimer : Copyright © 2020, Springer Nature.
Uncontrolled Keywords : Fine-grained; Sketch understanding; Image retrieval; Cross-modality; Deep learning
Depositing User : Clive Harris
Date Deposited : 07 Oct 2020 09:53
Last Modified : 07 Oct 2020 09:53

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800