The design of high-quality feature representation and efficient loss functions can boost the performance of object detection and other computer vision tasks. Currently, object detectors typically utilize FPN to enhance representation capabilities. However, FPN-based methods fail to fully exploit multi-scale features. In this paper, we propose the Adaptive Feature Pyramid Enhancement Network(AFPEN) to address this problem. Specifically, AFPEN consists of the Dilated Context Aggregation(DCA) module and the Weight-Adaptive Channel Integration (WACI) module, which are designed to capture richer contextual information. Besides, existing loss functions struggle to address object scale variations. To mitigate this, this paper introduces the Boundary Alignment IoU(BA-IoU) loss. BA-IoU improves the speed and accuracy of fitting predicted boxes to ground-truth boxes, particularly for objects of varying scales. The final evaluation of our method on COCO 2017 and Pascal VOC datasets shows competitive performance against state-of-the-art FPN-based detectors.