Joint Spatial and Layer Attention for Convolutional Networks

Tony Joseph (University of Ontario Institute of Technology), Konstantinos Derpanis (Ryerson University), Faisal Qureshi (University of Ontario Institute of Technology)

Abstract
In this paper, we propose a novel approach that learns to sequentially attend to different Convolutional Neural Networks (CNN) layers (i.e., ``what'' feature abstraction to attend to) and different spatial locations of the selected feature map (i.e., ``where'') to perform the task at hand. Specifically, at each Recurrent Neural Network step, both a CNN layer and localized spatial region within it are selected for further processing. We demonstrate the effectiveness of this approach on two computer vision tasks: (i) image-based six degrees of freedom camera pose regression and (ii) indoor scene classification. Empirically, we show that combining the ``what'' and ``where'' aspects of attention improves network performance on both tasks. We evaluate our method on standard benchmarks for camera localization (Cambridge, 7-Scenes, and TUM-LSI) and for scene classification (MIT-67 Indoor Scenes). For camera localization, our approach reduces the median error by 18.8% for position and 8.2% for orientation (averaged over all scenes), and for scene classification, it improves the mean accuracy by 3.4% over previous methods.

DOI
10.5244/C.33.43
https://dx.doi.org/10.5244/C.33.43

Files
Paper (PDF)
Supplementary material (PDF)

BibTeX