Learning Visual Actions Using Multiple Verb-Only Labels

Michael Wray (University of Bristol), Dima Damen (University of Bristol)

Abstract
This work introduces verb-only representations for both recognition and retrieval of visual actions. Current methods neglect legitimate semantic ambiguities between verbs, instead choosing unambiguous subsets of verbs along with objects to disambiguate the actions. By combining multiple verbs, through hard or soft assignment, as a regression, we are able to learn a much larger vocabulary of verbs, including contextual overlaps of these verbs. We collect multi-verb annotations for three action datasets and evaluate the verb-only labelling representations for action recognition and cross-modal retrieval (video-to-text and text-to-video). We demonstrate that multi-label verb-only representations outperform conventional single verb labels. We also explore other benefits of a multi-verb representation including cross-dataset retrieval and verb type (manner and result verb types) retrieval.

DOI
10.5244/C.33.123
https://dx.doi.org/10.5244/C.33.123

Files
Paper (PDF)

BibTeX