This paper presents a deep neural network (DNN) based fully spatiotemporal rain removal network, MoPE-Spatiotemporal, to enhance accuracy of activity recognition in rainy videos. The proposed network utilizes spatiotemporal information of an image sequence to detect rain streaks and recover the non-rainy image. We also present rain alert network that detects the rain fall and informs the reduction of recognition confidence under rain. Experimental results show that heavy rain can highly degrade activity recognition accuracy. MoPE-Spatiotemporal removes heavy rain better than state-of-the-art methods, and significantly improves (0.15) activity recognition accuracy in rainy videos with minimal impact on recognition accuracy in clean videos.