Forecasting Future Action Sequences with Neural Memory Networks

Harshala Gammulle (Queensland University of Technology), Simon Denman (Queensland University of Technology), Sridha Sridharan (Queensland University of Technology), Clinton Fookes (Queensland University of Technology)

Abstract
We propose a novel neural memory network based framework for future action sequence forecasting. This is a challenging task where we have to consider short-term, within sequence relationships as well as relationships in between sequences, to understand how sequences of actions evolve over time. To capture these relationships effectively, we introduce neural memory networks to our modelling scheme. We show the significance of using two input streams, the observed frames and the corresponding action labels, which provides different information cues for our prediction task. Furthermore, through the proposed method we effectively map the long-term relationships among individual input sequences through separate memory modules, which enables better fusion of the salient features. Our method outperforms the state-of-the-art approaches by a large margin on two publicly available datasets: Breakfast and 50 Salads.

DOI
10.5244/C.33.117
https://dx.doi.org/10.5244/C.33.117

Files
Paper (PDF)
Supplementary material (ZIP)

BibTeX