Deep learning based object detection architectures have significantly advanced the state of the art. However, a study of recent detection methods shows a wide gap between small object performance and performance on medium and large objects. This gap is prevalent across architectures and across backbones. We show that this gap is primarily due to reduction in the feature map size as we traverse the backbone. Through simple modifications to the backbone structure, we show a marked improvement in performance for small objects. In addition, we propose a dual-path configuration with weight sharing for recovering large object performance. Compared to state of the art methods that rely on multi-scale training and network partitioning we show competitive performance without any bells and whistles on the MS COCO dataset. We show state of the art small object performance with a mobile object detector SSD Mobilenet v1.