Attention-based Facial Behavior Analytics inSocial Communication

Lezi Wang (Rutgers University), Chongyang Bai (Dartmouth College), Maksim Bolonkin (Dartmouth College), VS Subrahmanian (Dartmouth College), Judee Burgoon (University of Arizona), Norah Dunbar (University of California, Santa Babara), Dimitris Metaxas (Rutgers University)

Abstract
In this study, we address a cross-domain problem of applying computer vision approaches to reason about human facial behavior when people play \textit{The Resistance} game. To capture the facial behaviors, we first collect several hours of video where the participants playing \textit{The Resistance} game assume the roles of deceivers (spies) vs truth-tellers (villagers). We develop a novel attention-based neural network (NN) that advances the state of the art in understanding how a NN predicts the players' roles. This is accomplished by discovering through learning those pixels and related frames which are discriminative and contributed the most to the NN's inference. We demonstrate the effectiveness of our attention-based approach in discovering the frames and facial Action Units (AUs) that contributed to the NN's class decision. Our results are consistent with the current communication theory on deception.

DOI
10.5244/C.33.98
https://dx.doi.org/10.5244/C.33.98

Files
Paper (PDF)

BibTeX