Generating Expensive Relationship Features from Cheap Objects

Xiaogang Wang (National University of Singapore), Qianru Sun (Singapore Management University), Marcelo Ang (National University of Singapore), Tat-Seng Chua (National University of Singapore)

Abstract
We investigate the problem of object relationship classification of visual scenes. For a relationship object1-predicate-object2 that captures the object interaction, its representation is composed by the combination of object1 and object2 features. As a result, relationship classification models usually bias to the frequent objects, leading to poor generalization to rare or unseen objects. Inspired by the data augmentation methods, we propose a novel Semantic Transform Generative Adversarial Network (ST-GAN) that synthesizes relationship features for rare objects, conditioned on the features from random instances of the objects. Specifically, ST-GAN essentially offers a semantic transform function from cheap object features to expensive relationship features. Here, “cheap” means any easy-to-collect object which possesses an original but undesired relationship attribute, e.g., a sitting person; “expensive” means a target relationship on this object, e.g., person-riding-horse. By generating massive triplet combinations from any object pair with larger variance, ST-GAN can reduce the data bias. Extensive experiments on two benchmarks – Visual Relationship Detection (VRD) and Visual Genome (VG), show that using our synthesized features for data augmentation, the relationship classification model can be consistently improved in various settings such as zero-shot and low-shot.

DOI
10.5244/C.33.133
https://dx.doi.org/10.5244/C.33.133

Files
Paper (PDF)
Supplementary material (ZIP)

BibTeX