With the advent of smart applications, computation technology is being brought to everyday devices. However, the rapid expansion of smart technology comes with a set of challenges. Smart devices have a limited computing capability, and usually rely on computing outsourcing architectures to externalize the computations, where Cloud servers become the main computing centers of the applications. As a result, computing infrastructure needs to be expanded to be able to cope with the ever-increasing demand of computations. To satisfy the demand for high performance computations, specialized computing devices are being included in cloud architectures. They boost the performance of artificial intelligence and high-performance computations, which are the basis of many smart society applications. The aim of this work is to provide a review on specialized computing device utilization in computing outsourcing architectures and the opportunities that it would bring to smart society applications. It also discusses the importance of specialized computing device virtualization in computing outsourcing architectures, and the challenges and opportunities that presents to increase the efficiency of these devices. Dealing with these challenges may lead to higher usage of specialized computing devices in the cloud and support the expansion of smart applications.