University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval

Dey, Sounak, Riba, Pau, Dutta, Anjan, Llados, Josep and Song, Yi-Zhe (2019) Doodle to Search: Practical Zero-Shot Sketch-based Image Retrieval In: 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR 2019), 16-20 Jun 2019, Long Beach, California, USA.

[img]
Preview
Text
CVPR2019.pdf - Accepted version Manuscript

Download (14MB) | Preview

Abstract

In this paper, we investigate the problem of zeroshot sketch-based image retrieval (ZS-SBIR), where human sketches are used as queries to conduct retrieval of photos from unseen categories. We importantly advance prior arts by proposing a novel ZS-SBIR scenario that represents a firm step forward in its practical application. The new setting uniquely recognizes two important yet often neglected challenges of practical ZS-SBIR, (i) the large domain gap between amateur sketch and photo, and (ii) the necessity for moving towards large-scale retrieval. We first contribute to the community a novel ZS-SBIR dataset, QuickDraw-Extended, that consists of 330; 000 sketches and 204; 000 photos spanning across 110 categories. Highly abstract amateur human sketches are purposefully sourced to maximize the domain gap, instead of ones included in existing datasets that can often be semi-photorealistic. We then formulate a ZS-SBIR framework to jointly model sketches and photos into a common embedding space. A novel strategy to mine the mutual information among domains is specifically engineered to alleviate the domain gap. External semantic knowledge is further embedded to aid semantic transfer. We show that, rather surprisingly, retrieval performance significantly outperforms that of state-of-the-art on existing datasets that can already be achieved using a reduced version of our model. We further demonstrate the superior performance of our full model by comparing with a number of alternatives on the newly proposed dataset. The new dataset, plus all training and testing code of our model, will be publicly released to facilitate future research.

Item Type: Conference or Workshop Item (Conference Paper)
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
NameEmailORCID
Dey, Sounak
Riba, Pau
Dutta, Anjan
Llados, Josep
Song, Yi-Zhey.song@surrey.ac.uk
Date : 2019
Copyright Disclaimer : © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Related URLs :
Depositing User : Diane Maxfield
Date Deposited : 03 Jun 2019 10:50
Last Modified : 04 Sep 2019 13:19
URI: http://epubs.surrey.ac.uk/id/eprint/851926

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year


Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800