Particularly in distributed systems, where data are spread across numerous devices or servers, privacy concerns have grown paramount in the age of big data and machine learning. A potential solution to these issues is federated learning (FL), which allows models to be trained on decentralised data without revealing sensitive information. Using federated learning as an example, this article provides a thorough overview of privacy-preserving machine learning methods for use in distributed systems. Assuring that sensitive data stay locally stored and encrypted during training, we go over the fundamentals of FL, which include federated optimization, model aggregation, and secure aggregation protocols. We also investigate several issues with FL systems; security, privacy, and communication overhead, as well as potential solutions to these problems. Case studies in various domains, including healthcare, finance, and the Internet of Things (IoT), demonstrate how FL effectively preserves privacy. Lastly, we draw attention to possible future paths for privacy-preserving ML research and development, stressing the need for strong cryptographic methods, effective communication protocols, and standardized frameworks to implement FL in distributed systems in the real world.