University of Surrey

Test tubes in the lab Research in the ATI Dance Research

Visual narratives : free-hand sketch for visual search and navigation of video.

James, Stuart (2016) Visual narratives : free-hand sketch for visual search and navigation of video. Doctoral thesis, University of Surrey.

[img]
Preview
Text
James2015_Visual_Narratives_Free-Hand_Sketch_for_Visual_Search_and_Navigation_of_Video.pdf - Version of Record
Available under License Creative Commons Attribution Non-commercial Share Alike.

Download (61MB) | Preview
[img]
Preview
Text
Author_Deposit_Agreement.pdf - Version of Record
Available under License Creative Commons Attribution Non-commercial Share Alike.

Download (165kB) | Preview

Abstract

Humans have an innate ability to communicate visually; the earliest forms of communication were cave drawings, and children can communicate visual descriptions of scenes through drawings well before they can write. Drawings and sketches offer an intuitive and efficient means for communicating visual concepts. Today, society faces a deluge of digital visual content driven by a surge in the generation of video on social media and the online availability of video archives. Mobile devices are emerging as the dominant platform for consuming this content, with Cisco predicting that by 2018 over 80% of mobile traffic will be video. Sketch offers a familiar and expressive modality for interacting with video on the touch-screens commonly present on such devices. This thesis contributes several new algorithms for searching and manipulating video using free-hand sketches. We propose the Visual Narrative (VN); a storyboarded sequence of one or more actions in the form of sketch that collectively describe an event. We show that VNs can be used to both efficiently search video repositories, and to synthesise video clips. First, we describe a sketch based video retrieval (SBVR) system that fuses multiple modalities (shape, colour, semantics, and motion) in order to find relevant video clips. An efficient multi-modal video descriptor is proposed enabling the search of hundreds of videos in milliseconds. This contrasts with prior SBVR that lacks an efficient index representation, and take minutes or hours to search similar datasets. This contribution not only makes SBVR practical at interactive speeds, but also enables user-refinement of results through relevance feedback to resolve sketch ambiguity, including the relative priority of the different VN modalities. Second, we present the first algorithm for sketch based pose retrieval. A pictographic representation (stick-men) is used to specify a desired human pose within the VN, and similar poses found within a video dataset. We use archival dance performance footage from the UK National Resource Centre for Dance (UK-NRCD), containing diverse examples of human pose. We investigate appropriate descriptors for sketch and video, and propose a novel manifold learning technique for mapping between the two descriptor spaces and so performing sketched pose retrieval. We show that domain adaptation can be applied to boost the performance of this system through a novel piece-wise feature-space warping technique. Third, we present a graph representation for VNs comprising multiple actions. We focus on the extension of our pose retrieval system to a sequence of poses interspersed with actions (e.g. jump, twirl). We show that our graph representation can be used for multiple applications: 1) to retrieve sequences of video comprising multiple actions; 2) to navigate in pictorial form, the retrieved video sequences; 3) to synthesise new video sequences by retrieving and concatenating video fragments from archival footage.

Item Type: Thesis (Doctoral)
Divisions : Theses
Authors :
AuthorsEmailORCID
James, Stuarts.james@surrey.ac.ukUNSPECIFIED
Date : 29 February 2016
Funders : Centre for Vision Speech and Signal Processing
Contributors :
ContributionNameEmailORCID
Thesis supervisorCollomosse, Johnj.collomosse@surrey.ac.ukUNSPECIFIED
Depositing User : Stuart James
Date Deposited : 01 Mar 2016 10:35
Last Modified : 01 Mar 2016 10:35
URI: http://epubs.surrey.ac.uk/id/eprint/809807

Actions (login required)

View Item View Item

Downloads

Downloads per month over past year


Information about this web site

© The University of Surrey, Guildford, Surrey, GU2 7XH, United Kingdom.
+44 (0)1483 300800