Polyps – indicators of lethal colorectal cancer and the focus of early screening and prevention, are missed to varying degrees in clinics due to morphological differences. Existing methods either need annotated bounding boxes, neglect the reality of unprepared polyp proposals, or lack complete predictions. Even worse, they only focus on the detection rate of polyps under pathological classification. To overcome these issues, we creatively propose the Morphology-Driven network (MDNet), which detects polyps with only image-level supervision. Specifically, by thinking of the generic feature between detection and segmentation, the cross-domain reference module (CRM) is devised to decrease the negative effect of the uncertain proposals. Based on spatial differences in polyp morphologies, the spatial category module (SCM) is designed, which enhances the ability to discriminate similar polyps of different morphology. In addition, class and region scores are integrated into the dual-threshold post-processing strategy (DPS) to improve detection accuracy. We carry out the experiments on three datasets (one internal and two public) and experimental results indicate that MDNet has better robustness and performance. All code is available at https://github.com/dxqllp/MDNet .