Fast Zero-Shot Image Tagging

A tagging method that is

Zero-shot: assign to query image both seen & unseen tags (unseen at training)

Fast: O(n) training, O(1) testing; tag thousands of images in seconds

Accurate: state-of-the-art on 2 datasets, 3 tagging tasks, & various metrics

Code & Toy example

Note: Keras & Theano  are required

A conventional tagging toy example on IAPR TC-12 dataset can be downloaded here. See included readme for more details.

Approach

 

Given an image, its relevant tag’s word vectors rank ahead of the irrelevant tag’s along some direction in the word vector space. We call that direction the principal direction for the image. To solve the problem of image tagging, we thus learn a function  to approximate the principal direction from an image. This function takes as the input an image  and outputs a vector  for defining the principal direction in the word vector space.

Presentation

Related Publication

Please refer to following papers if you want to cite our work in your publication:

Fast zero-shot image tagging

@InProceedings{Zhang_2016_CVPR,
author = {Zhang, Yang and Gong, Boqing and Shah, Mubarak},
title = {Fast Zero-Shot Image Tagging},
booktitle = {The IEEE Conference on Computer Vision and Pattern
Recognition (CVPR)},
month = {June},
year = {2016}
}
Infinite-Label Learning with Semantic Output Codes

@misc{1608.06608,
Author = {Yang Zhang and Rupam Acharyya and Ji Liu and
Boqing Gong},
Title = {Infinite-Label Learning with Semantic Output Codes},
Year = {2016},
Eprint = {arXiv:1608.06608},
}