Numerous learning-based motion planning techniques have been put forth in the literature for autonomous driving. These techniques can directly predict motion commands from the sensory data of the environment, but they are unable to predict multiple motion commands, such as steering angle, accelerator, and brake, or balance errors between various motion commands. This study examines the simulation outcomes of an autonomous vehicle learning to operate in a streamlined environment with just static impediments and lane lines. This review offers a thorough examination of several threats that could endanger autonomous driving systems (ADS), as well as the associated cutting-edge defense techniques. The research begins by providing a comprehensive review of each stage of the ADS workflow, covering adversarial assaults for various deep learning models and attacks in both physical and virtual environments. These assaults inevitably pose a serious threat to the safety and security of deep learning-based autonomous driving, from which the remedies should be thoroughly researched and investigated to reduce any potential hazards. This review offers a detailed examination of several threats that might endanger ADSs, as well as the associated cutting-edge protection techniques. The adversarial assaults, which restrict the applications’ performance, might target certain tensor perturbations in machine learning models. Implementing defensive models against adversarial assaults is thus a crucial research subject nowadays. For the purpose of enhancing the safety of deep learning-based autonomous driving, certain intriguing research avenues are also recommended. The various defense schemes have also been illustrated in this paper.