Fusion-based pedestrian detection is crucial for environment perception in unmanned driving. However, existing fusion-based frameworks face challenges such as slow processing speeds, high training costs, and difficulties in deployment on Unmanned Ground Vehicles (UGVs). To address these issues, this study proposes a real-time decision-level fusion detection framework based on Intersection over Union (IoU) matching, leveraging the highly real-time YOLO-P model as the primary detector. By exploiting the pose relationship between the camera and LiDAR, we extract point cloud pedestrian regions of interest (ROIs) to reduce computation and use a real-time clustering technique to group multiple pedestrians with small intervals within these ROIs. By leveraging 3D bounding boxes derived from point cloud data and matching them with 2D bounding boxes obtained from image analysis, the confidence level of pedestrian detection is dynamically adjusted to significantly enhance detection accuracy. Experiments conducted on a dataset collected within a campus environment demonstrate that our proposed fusion strategy achieves excellent performance in challenging pedestrian scenes characterized by narrow intervals, severe occlusion, and distant positions. Furthermore, the strategy exhibits outstanding real-time capabilities.