We propose a fast and accurate system for automatically estimating the quality of food grains on resource constrained portable devices using computer vision. We are motivated by an urgent need in India for grain quality estimation to ensure transparency in the agricultural supply chain and empower poor farmers to get the correct price for their crops. The system uses instance segmentation of touching grains, followed by classification of each grain according to E-NAM parameters. To the best of our knowledge, this is the first attempt to use Deep Learning to estimate quality of cluttered sample of grains using only mobile phone. Samples are collected from various Agricultural Produce Market Committee (APMC) yards, which are used to generate synthetic data to simulate realistic clutter of grains for training our instance-segmentation network. Novel augmentation techniques while training make the system robust to illumination changes. Our system obtains the state-of-the-art performance and has been tested in various locations in India. At a mAP score of 0.74 and classification accuracy 92%, our system takes less than 100s compared to 15 minutes of manual quality estimation.