We propose a novel local diffusion aided single stage detector for radar to tackle the noise and sparsity issues of its point cloud in detection tasks. We utilized space occupancy maps to represent the downsampled point cloud and applied a diffusion module to denoise them. We also introduced instance-aware downsampling strategies and 3D upsampling module to enhance the model’s perception ability in 3D space. Experiments show that LDRadSSD outperformed those SOTA approaches in predicting bounding boxes for road users such as cyclists and pedestrians and figuring out the drivable space in bird’s eye view (BEV) of the scene. In particular, with IoU thresholds of 0.5/0.25/0.25, the average prediction precision (AP) of main type of road users (cars, pedestrians and cyclists) reached at competitive 58.5%, 63.1%, and 82.1%, respectively, while mean IoU of free space was 92.6%. Moreover, the prediction precision of object orientation dramatically raised to averaged 71.1%. We also demonstrate in this paper that LDRadSSD can satisfy real-time requirements in autonomous driving as its running speed reach at 18.2 to 41.7 milliseconds per frame.