Fine-Grained Instance-Level Sketch-Based Image Retrieval

Yu, Qian, Song, Jifei, Song, Yi-Zhe, Xiang, Tao and Hospedales, Timothy M. (2020) Fine-Grained Instance-Level Sketch-Based Image Retrieval International Journal of Computer Vision.

Full text not available from this repository.

Abstract

The problem of fine-grained sketch-based image retrieval (FG-SBIR) is defined and investigated in this paper. In FG-SBIR, free-hand human sketch images are used as queries to retrieve photo images containing the same object instances. It is thus a cross-domain (sketch to photo) instance-level retrieval task. It is an extremely challenging problem because (i) visual comparisons and matching need to be executed under large domain gap, i.e., from black and white line drawing sketches to colour photos; (ii) it requires to capture the fine-grained (dis)similarities of sketches and photo images while free-hand sketches drawn by different people present different levels of deformation and expressive interpretation; and (iii) annotated cross-domain fine-grained SBIR datasets are scarce, challenging many state-of-the-art machine learning techniques, particularly those based on deep learning. In this paper, for the first time, we address all these challenges, providing a step towards the capabilities that would underpin a commercial sketch-based object instance retrieval application. Specifically, a new large-scale FG-SBIR database is introduced which is carefully designed to reflect the real-world application scenarios. A deep cross-domain matching model is then formulated to solve the intrinsic drawing style variability, large domain gap issues, and capture instance-level discriminative features. It distinguishes itself by a carefully designed attention module. Extensive experiments on the new dataset demonstrate the effectiveness of the proposed model and validate the need for a rigorous definition of the FG-SBIR problem and collecting suitable datasets.

Item Type:	Article
Divisions :	Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :	Name Email ORCID Yu, Qian Song, Jifei Song, Yi-Zhe y.song@surrey.ac.uk Xiang, Tao t.xiang@surrey.ac.uk Hospedales, Timothy M.
Date :	30 September 2020
DOI :	10.1007/s11263-020-01382-3
Copyright Disclaimer :	Copyright © 2020, Springer Nature.
Uncontrolled Keywords :	Fine-grained; Sketch understanding; Image retrieval; Cross-modality; Deep learning
Depositing User :	Clive Harris
Date Deposited :	07 Oct 2020 09:53
Last Modified :	07 Oct 2020 09:53
URI:	http://epubs.surrey.ac.uk/id/eprint/858692

Actions (login required)

View Item

Downloads