The growing application of drones is urging the development of drone-based object detection technologies, particularly in the areas of security, rescue, etc. However, these scenarios frequently involve extremely complex background that interferes with the real objects, which poses a significant challenge and limits the performance of object detection. To this end, an effective High Resolution Dual Branch Network (HRDBNet) is proposed for object detection in complex scenarios. Specifically, to address the problem of missed detections of small objects due to the scarcity of appearance features, a cross-layer high-resolution feature extraction network is designed to enhance the utilization of features for small objects. Meanwhile, in order to mitigate the complex background interference, a dual-branch structure is devised to detect object and background separately. Additionally, contrastive loss is employed to incrementally increase differentiation between object and background features, thereby achieving background suppression. Most impressively, on the challenging CS-Drone dataset, HRDBNet reaches 43.4% in AP, outperforming the existing state-of-the-art detectors, which indicates that the proposed method achieves robust object detection for drone-captured complex scenarios.