Object detection is one of the crucial branches of computer vision areas, which is generally utilized to identify and localize the spatial orientation of individual objects within a series of provided images or video streams. In this paper, we propose a novel object detection architecture named MS-Faster R-CNN based on the mechanism of multi-scale feature fusion and the backbone of Faster R-CNN. The fusion strategy mainly uses the structure of the Feature Pyramid Network (FPN), incorporating two links to combine the feature fusion, which enriches the semantics of the fused features and is suitable for various scales of objects. The cascaded Region Proposal Network(RPN) along with the optimized Non-Maximum Suppression (NMS) algorithm are utilized in the candidate box recommendation stage to overcome the problem of over-suppression of small-scale objects in region candidate boxes. The recommendation efficiency of the candidate box can be greatly improved. Finally, the Region of Interest (ROI) Align pooling technology based on the bilinear interpolation method is considered to avoid the loss of accuracy caused by quantization.