A Two-Stage Model for 3D Object Recognition and Classification in Autonomous Vehicles: Camera and Lidar Fusion Method

3D object recognition and planning are two essential and closely related components in autonomous vehicles. Accurate 3D object detection and classification contribute to improved trajectory estimation and prediction of moving objects, enhancing the effectiveness of the planning module. However, the object classification capability of Lidar has its limitations, making it beneficial to integrate visual information from cameras for more precise object classification. Additionally, classifying the states of vehicles, such as braking, turning left, or turning right, further refines the operation of the planning module in autonomous vehicles. In this paper, we propose a two-stage model to combine information from Lidar and cameras for 3D object recognition and vehicle state classification. The proposed model consists of the detection stage and the recognition stage. In the detection stage, we employ a deep learning model to process point cloud data from Lidar to determine 3D bounding boxes of objects. In the recognition stage, we project the 3D bounding boxes onto images captured by cameras to generate 2D bounding boxes, which are then fed into a deep learning classifier for object and state identification if applicable. To train the model, we use the publicly available KITTI dataset and introduce an additional set of 20 classes to account for vehicle states such as ‘break,’ ‘turn left,’ ‘turn right,’ and ‘off’ for Car, Van, Truck, Tram, and Misc. Experimental results demonstrate the accuracy and effectiveness of our proposed 3D object recognition method for both detection and classification tasks.