Ptychography is a coherent diffractive imaging (CDI) method that captures multiple diffraction patterns of a sample with a set of shifted localized illuminations (``probes''). The reconstruction problem, known as ``phase retrieval'', is conventionally solved by iterative algorithms. In this paper, we propose PtychoNet, a deep learning based method to perform phase retrieval for ptychography in a non-iterative manner. We devise a generative network to encode a full ptychography scan, reverse the diffractions at each scanning point and compute the amplitude and phase of the object. We demonstrate successful reconstruction using PtychoNet as well as recovering fine features in the case of extreme sparse scanning where conventional methods fail to give recognizable features.