The advent of intelligent video analytics systems has changed the conventional method of video data processing and analysis, which with its implementation has enabled applications in different domains like security and surveillance, healthcare, self-driving vehicles, and smart cities. One of the leading facets of artificial intelligence, deep learning, currently stands out as a highly disruptive technique for solving large classes of difficult problems, including challenging high-dimensional video interpretation tasks. This survey presents a systematic overview that covers recent developments and techniques for designing skillful video analytics systems by utilizing deep learning in general. It discusses important methods, and applications, and emphasizes the problems of integrating these systems and future works to be carried out in this domain. The research is not only focused on video analytics systems (e.g. activity recognition, and object detection) and real-time aspects but also highlights ethical and practical considerations. This survey summarizes up-to-date advances from the technical literature and will hopefully provide a useful reference for researchers and practitioners wishing to develop better intelligent video analytics systems.