Visual Perception Stack for Autonomous Vehicle

Visual perception stack is indispensable for present-day autonomous vehicle. It perceives the environment around the ego vehicle in the same way as humans. Stereo cameras are the prominent sensor for visual perception stack. This paper will focus on deep-level understanding of visual perception using stereo camera, image processing, and crucial aspects required for autonomous cars. There are other sensors such as LIDAR, GNSS, and IMU, which are used for environment perception, but the information from stereo cameras are far profitable and utilitarian than other sensor information. First, the image formation phenomenon is discussed followed by the image projection onto different frames such as world frame, camera frame, image coordinates, and pixel coordinate. Then camera calibration will be discussed and intrinsic parameters of stereo camera are obtained from RQ factorization method. Depth perception from stereo camera is done by using identifying epipolar line and by arriving at disparity map, depth map, and finally the cross correlation. Then feature detection, feature description, and feature matching are essential in order to establish a robust image detection. The algorithms involved in each part will be discussed along with outlier rejection using RANSAC algorithm. In order to obtain a fine image detection, CNN will be used along with pooling layer and feature decoder to up sample the image. The output of semantic segmentation will be obliging for drivable space estimation, object detection, and distance-to-collision on environment.