The domain of Artificial Intelligence (AI) is a much-debated and highly productive field for new technological innovations and applications. While AI has for several years now been used to develop products and services that are core to many industries, it is in the healthcare sector that the promise of AI seems to be at its greatest. AI-enabled devices promise to improve patient outcomes, compress decision timeframes, enhance patient and practitioner experiences, and lower costs by expediting and improving healthcare delivery. Healthcare is a unique domain with multiple pillars: Health Systems, Health Insurance, Patients, and Healthcare Providers. The activities within and coordination between these pillars generate huge amounts of data. Advances in Natural Language Processing, Machine Learning, and AI/ML approaches such as Convolutional Neural Networks, Deep Learning, and Reinforcement Learning, among many others, provide a sophisticated analytical lens on these data. From advanced drug discovery to improved diagnoses and care delivery, AI has the potential to optimize, automate, and improve multiple healthcare processes. However, this increasing reliance of the healthcare sector on AI brings with it a host of challenges. Given that AI relies on data to learn and produce outcome predictions, it is the ethical and responsible usage of the data itself that warrants scrutiny. This is particularly true with respect to data privacy. The response to a data privacy-related concern is usually in the form of regulations that pose strict limits on how, when, where, and by whom data can be utilized. The healthcare industry, in particular, is burdened with a raft of compliance mandates that require adherence to specific data privacy policies. While the rise of AI in the healthcare sector has prompted the development of several innovative products and services, it has also created a decidedly contradictory environment, with several startups—whose success depends on the wise usage of massive amounts of sensitive data—working in the very domain that is itself likely to exert the strictest restrictions on data usage.