<p>Safety and security in Autonomous Vehicles (AVs) are greatly dependent on perception. To get an understanding of the surrounding environment, feed sensor data to the perception algorithm. For 3D object detection tasks, Multi-Sensor Fusion (MSF)-based perception is gaining popularity. Even though, there have been several relevant work studies on 3D object detection issues, all of them use MSF design-based AVs perception to detect objects using either LiDAR, camera, or both and consider MSF design to be robust against attacks. Nevertheless, some of the previous works attack MSF design and encourage risky and incorrect decision-making. In this paper, we address the security issue of MSF design-based perception in AVs. We develop the architecture of MSF-based perception by adding a segmentation part to 2D images collected via a single model (camera modality) before perception. This MSF design, which concentrates on a single model, is viewed as less significant than other MSF designs employed in previous works. We proposed a robust Multi-Sensor Fusion approach-based segmentation to defend against adversarial patch attacks introduced into a single model (camera modality). Our approach employs two stages first, detect objects comprehensively using lidar point cloud and RGB images to estimate 3D bounding boxes; second, it evaluates the robustness of the proposed approach against adversarial patch attacks. Our experimental evaluation shows that our proposed approach significantly outperforms all the state-of-the-art. It achieves superior performance in 3D object detection, as demonstrated by its accuracy of 95.06%, and defends against adversarial patch attacks, achieving an accuracy of 94.7% for partially covered objects and 94.4% for completely covering objects respectively on the challenging KITTI benchmark.</p>