University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval

Sain, Aneeshan, Bhunia, Ayan Kumar, Yang, Yongxin, Xiang, Tao and Song, Yi-Zhe (2020) Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval In: The 31st British Machine Vision Virtual Conference (BMVC 2020), 07-10 Sep 2020, Online (virtual).

Cross-Modal Hierarchical Modelling for Fine-Grained Sketch Based Image Retrieval - VoR.pdf - Version of Record

Download (4MB) | Preview


Sketch as an image search query is an ideal alternative to text in capturing the finegrained visual details. Prior successes on fine-grained sketch-based image retrieval (FGSBIR) have demonstrated the importance of tackling the unique traits of sketches as opposed to photos, e.g., temporal vs. static, strokes vs. pixels, and abstract vs. pixelperfect. In this paper, we study a further trait of sketches that has been overlooked to date, that is, they are hierarchical in terms of the levels of detail – a person typically sketches up to various extents of detail to depict an object. This hierarchical structure is often visually distinct. In this paper, we design a novel network that is capable of cultivating sketch-specific hierarchies and exploiting them to match sketch with photo at corresponding hierarchical levels. In particular, features from a sketch and a photo are enriched using cross-modal co-attention, coupled with hierarchical node fusion at every level to form a better embedding space to conduct retrieval. Experiments on common benchmarks show our method to outperform state-of-the-arts by a significant margin.

Item Type: Conference or Workshop Item (Conference Paper)
Divisions : Faculty of Engineering and Physical Sciences > Electronic Engineering
Authors :
Bhunia, Ayan
Date : 2020
Copyright Disclaimer : © BMVC 2020. The copyright of this document resides with its authors. It may be distributed unchanged freely in print or electronic forms.
Related URLs :
Depositing User : Clive Harris
Date Deposited : 07 Oct 2020 09:39
Last Modified : 07 Oct 2020 09:39

Actions (login required)

View Item View Item


Downloads per month over past year

Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800