<p>Visual Object Tracking (VOT) aims to track one or multiple targets in a video sequence based on the object description. In the past decade, the development of trackers has been one of the most important fields in machine vision. Before implementing an existing tracker or proposing a new one, developers must have a comprehensive knowledge of the datasets, trackers, evaluation protocols and the comparison between various trackers in VOT. Although VOT researchers have presented some review papers; this field still lacks a complete reference covering all VOT areas. In the present review paper, all domains of tracking are covered, including color (RGB), Thermal Infrared (TIR or T), Depth (D), Event (E), RGB-T, RGB-D, and RGB-E in various categories such as datasets, evaluation criteria, and trackers. So far, this comprehensive review is the most extensive one in the area of VOT. For this purpose, 64 diverse Single-Object-Tracking (SOT) and Multi-Object-Tracking (MOT) datasets (TIR:5, RGB:35, D:1, RGB-T:11, RGB-D: 9 and RGB-E: 6) are introduced and reviewed, which are selected based on their utilization by researchers. Also, all evaluation criteria available for SOT and MOT purposes are presented in this work. Finally, 216 different trackers (Thermal:11, RGB:126, RGB-T:35, RGB-D:27, RGB-E: 12, Depth: 5) are surveyed and a general comparison between them is made in various modalities of VOT to show their strengths and weaknesses. These trackers are categorized based on their main techniques including deep or traditional methods. Furthermore, the access links to mentioned datasets and trackers are provided in this paper.</p>