Machine learning (ML), a subfield of artificial intelligence, holds immense applications across various domains owing to its potential to analyze vast datasets and discern intricate patterns. By leveraging algorithms and statistical models, ML facilitates the extraction of actionable insights and predicts future outcomes based on historical data. This predictive capability underpins a multitude of applications, spanning computer vision, natural language processing, finance, healthcare, transportation, wireless communications, and beyond. Through sophisticated algorithms, such as neural networks (NNs), ML empowers systems to autonomously adapt and refine their behavior based on experience, driving innovation and efficiency in diverse fields. Consequently, ML is a cornerstone of modern technological advancements, offering unparalleled opportunities for data-driven decision-making, process optimization, and transformative breakthroughs. ML has been successful and is envisioned to play a pivotal role in revolutionizing wireless communication systems by offering innovative solutions to enhance system performance and efficiency, reliability, and security. With the proliferation of wireless devices and the exponential growth in data traffic, conventional communication system techniques face unprecedented challenges, such as managing network resources, complex wireless networks, unknown physical and signal models, mitigating interferences, and ensuring seamless connectivity. In Sect. 11.1, we discuss different ML-based solutions developed for wireless communications. Specifically, we have considered two classification-based applications: signal detection and modulation classification of wireless signals, and two regression-based applications: wireless channel state information estimation and prediction of optimal power allocation. It was established that the integration of ML into wireless communication systems promises to unlock new capabilities, improve network performance, and pave the way for next-generation wireless technologies. However, one key challenge in deploying ML systems in real-world applications is the model complexity and high computational requirements. The edge devices, such as smartphones, drones, and Internet-of-Things systems, are limited by storage capacity and computational power. Therefore, optimizing the highly complex ML and deep learning models for efficient deployment in real-world edge applications is essential. In Sect. 11.2, we discuss three different techniques for model optimization, i.e., network pruning, model quantization, and knowledge distillation. For demonstration purposes, modulation recognition of wireless signals is considered the application of interest, and the effectiveness of the three methods to optimize the complex models for edge applications without compromising performance is established. Specifically, we evaluate the performance of the optimized ML methods in terms of sparsity, storage compression, and reduction in the number of parameters. We experimentally show that the network pruning method is beneficial to achieve high sparsity by removing the unimportant parameters of the ML model, and quantization helps in representing the model parameters using a smaller number of bits, e.g., 8 bits compared to the original high-bit representations, e.g., 32 or 64 bits. Finally, the knowledge distillation technique effectively transfers features learned by a large and complex ML model to a smaller model (i.e., reduced number of model parameters), which can help the smaller model achieve a performance comparable to the large model. The successful emergence of ML in wireless communication applications has raised significant concerns about security-related challenges. Among many security challenges, one such is adversarial attacks. Section 11.3 presents the susceptibility of ML-based models against adversarial attacks for classification- and regression-based wireless communication applications. Firstly, we consider the joint problem of developing optimized ML models and robustification of these models against adversarial attacks, thus enabling efficient and reliable deployment of ML-based modulation classification of radio signals on edge devices. We propose utilizing two optimized models, knowledge distillation and network pruning, followed by computationally efficient adversarial training as a countermeasure technique against adversarial attacks to improve the robustness. Through experiments, we show that the proposed techniques significantly improve robustness when the ML models are optimized and adversarially trained compared to the unoptimized models. Secondly, we show that adversarial attacks can predict infeasible solutions for a regression problem in a wireless setting, that is, downlink optimal power allocation in a multi-cell massive multiple-input–multiple-output system. Further, we demonstrate the effectiveness of adversarial training as a countermeasure technique against these adversarial attacks and show that the robustness of DL-based wireless systems for regression application can significantly improve. We analyze the performance of the deep neural network models against adversarial attacks by considering the crafting of adversarial perturbations in both white-box and black-box manner.