TAGAN: Tonality Aligned Generative Adversarial Networks for Realistic Hand Pose Synthesis

Liangjian Chen (University of California, Irvine), Shih-Yao Lin (Tencent Medical AI Lab), Yusheng Xie (Tencent Medical AI Lab), Hui Tang (Tecent Medical AI Lab), Yufan Xue (workday), Yen-Yu Lin (Academia Sinica), Xiaohui Xie (University of California, Irvine), Wei Fan (Tencent)

Abstract
Despite recent progress, estimating 3D hand poses from single RGB images remains challenging. One of the major limiting factors is the lack of sufficiently large hand pose datasets with accurate 3D hand keypoint annotations. To address this limitation, we present an efficient method for generating realistic hand poses, and show that existing algorithms for hand pose estimation can be greatly improved by augmenting training data with images of the synthetic hand poses, which come naturally with ground truth annotations. More specifically, we adopt an augmented reality simulator to synthesize hand poses with accurate 3D hand-keypoint annotations. However, these synthesized hand poses look unnatural. To produce more realistic hand poses, we propose to blend each synthetic hand pose with a real background. To this end, we develop tonality-aligned generative adversarial networks (TAGAN), which align the tonality and color distributions between synthetic hand poses and real backgrounds, and can generate high-quality hand poses. TAGAN is evaluated on the RHP, STB, and CMU-PS hand pose datasets. With the aid of the synthesized poses, our method performs favorably against the state-of-the-arts in both 2D and 3D hand pose estimation.

DOI
10.5244/C.33.81
https://dx.doi.org/10.5244/C.33.81

Files
Paper (PDF)

BibTeX