An Efficient 3D CNN for Action/Object Segmentation in Video

Rui Hou (UCF), Chen Chen (University of North Carolina at Charlotte), Mubarak Shah (University of Central Florida), Rahul Sukthankar (Google)

Abstract
Convolutional Neural Network (CNN) based image segmentation has made great progress in recent years. However, video object segmentation remains a challenging task due to its high computational complexity. Most of the previous methods employ a two-stream CNN framework to handle spatial and motion features separately. In this paper, we propose an end-to-end encoder-decoder style 3D CNN for video object segmentation. The proposed approach leverages 3D separable convolutions and drastically reduces the number of trainable parameters. Instead of the popular two-stream framework, we adopt 3D CNN to aggregate spatial and temporal information. To efficiently process video, we propose 3D separable convolution for the pyramid pooling module and decoder, which dramatically reduces the number of operations while maintaining the performance. Additionally, we also extend our framework to video action segmentation by adding an extra classifier to predict the action label for actors in videos. Extensive experiments on several video datasets demonstrate the superior performance of the proposed approach for action and object segmentation compared to the state-of-the-art.

DOI
10.5244/C.33.25
https://dx.doi.org/10.5244/C.33.25

Files
Paper (PDF)

BibTeX