This paper proposes an ensemble approach to real-time object detection that combines the strengths of YOLOv5 and Faster R-CNN. YOLOv5 is used to generate a set of object proposals, then Faster R-CNN is used to refine these proposals and produce more accurate object detection results. To further improve the accuracy of our object detection results, a cascade refinement network that uses dynamic fine-tuning is utilized, which uses the Kullback-Leibler divergence to dynamically adjust the weights of the Faster R-CNN model based on the confidence scores of the YOLOv5 object proposals. Our approach is evaluated on a novel dataset collected in Uganda with other State-of-the-art approaches which include RetinaNet, Cascade R-CNN, Single-Shot MultiBox Detector, and R-CNN. Experimental results revealed that our proposed ensemble model outperforms both base models with an average precision of 0.96, which is significantly higher than that of YOLOv5 (0.91) and Faster R-CNN (0.90). Our model was also able to achieve real-time inference speeds, processing frames at a rate of 25 frames per second, the same speed achieved by YOLOv5, faster than the speed of 15 frames per second by Faster R-CNN. Our results also revealed that the proposed ensemble model is comparable to other state-of-the-art object detection models. Our proposed approach can be used to improve the accuracy and speed of real-time object detection in a variety of applications.